diff --git a/README.ja.md b/README.ja.md index 92034884..d4662638 100644 --- a/README.ja.md +++ b/README.ja.md @@ -68,7 +68,6 @@ Vulsのセットアップは以下の3パターンがある - Dockerコンテナ上にセットアップ see https://github.com/future-architect/vuls/tree/master/setup/docker -[日本語README](https://github.com/future-architect/vuls/blob/master/setup/docker/README.ja.md) - Chefでセットアップ see https://github.com/sadayuki-matsuno/vuls-cookbook - 手動でセットアップ @@ -76,13 +75,12 @@ Hello Vulsチュートリアルでは手動でのセットアップ方法で説 ---- -# Hello Vuls +# Tutorial: Local Scan Mode 本チュートリアルでは、Amazon EC2にVulsをセットアップし、自分に存在する脆弱性をスキャンする方法を説明する。 手順は以下の通り 1. Amazon Linuxを新規作成 -1. 自分自身にSSH接続できるように設定 1. 必要なソフトウェアをインストール 1. go-cve-dictionaryをデプロイ 1. Vulsをデプロイ @@ -105,21 +103,7 @@ Hello Vulsチュートリアルでは手動でのセットアップ方法で説 - [Q: How do I disable the automatic installation of critical and important security updates on initial launch?](https://aws.amazon.com/amazon-linux-ami/faqs/?nc1=h_ls) -## Step2. SSH setting - -ローカルホストにSSH接続できるようにする。 - -SSHキーペアを作成し、公開鍵をauthorized_keysに追加する。 -```bash -$ ssh-keygen -t rsa -$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys -$ chmod 600 ~/.ssh/authorized_keys -``` - -VulsはSSHパスワード認証をサポートしていない。SSH公開鍵鍵認証を使う必要がある。 -また、パスワードありのSUDOもセキュリティ上の理由によりサポートしていないため、スキャン対象サーバに/etc/sudoersにNOPASSWDを設定して、パスワードなしでSUDO可能にする必要がある。 - -## Step3. Install requirements +## Step2. Install requirements Vulsセットアップに必要な以下のソフトウェアをインストールする。 @@ -150,7 +134,7 @@ export PATH=$PATH:$GOROOT/bin:$GOPATH/bin $ source /etc/profile.d/goenv.sh ``` -## Step4. Deploy [go-cve-dictionary](https://github.com/kotakanbe/go-cve-dictionary) +## Step3. Deploy [go-cve-dictionary](https://github.com/kotakanbe/go-cve-dictionary) ```bash $ sudo mkdir /var/log/vuls @@ -177,7 +161,14 @@ $ ls -alh cve.sqlite3 -rw-r--r-- 1 ec2-user ec2-user 7.0M Mar 24 13:20 cve.sqlite3 ``` -## Step5. Deploy Vuls +日本語化したい場合は、JVNから脆弱性データベースを取得する。 + +```bash +$ cd $HOME +$ for i in `seq 1998 $(date +"%Y")`; do go-cve-dictionary fetchjvn -years $i; done +``` + +## Step4. Deploy Vuls 新規にターミナルを起動し、先ほど作成したEC2にSSH接続する。 ``` @@ -188,44 +179,32 @@ $ cd vuls $ make install ``` -vulsを既にインストール済みでupdateしたい場合は - -```bash -$ go get -u github.com/future-architect/vuls -``` - -で可能である。 - -go getでエラーが発生した場合は、以下の点を確認する。 -- Gitのバージョンがv2以降か? -- Go依存パッケージの問題でgo getに失敗する場合は [deploying with glide](https://github.com/future-architect/vuls/blob/master/README.md#deploy-with-glide) を試す。 - -## Step6. Config +## Step5. Config Vulsの設定ファイルを作成する(TOMLフォーマット) -設定ファイルのチェックを行う ``` $ cd $HOME $ cat config.toml [servers] -[servers.172-31-4-82] -host = "172.31.4.82" -port = "22" -user = "ec2-user" -keyPath = "/home/ec2-user/.ssh/id_rsa" - +[servers.localhost] +host = "localhost" +port = "local" ``` -## Step7. Check config.toml and settings on the server before scanning +Root権限が必要なディストリビューションもあるで、スキャン対象サーバの/etc/sudoersを変更する。 +パスワードありのsudoはセキュリティ上の理由からサポート指定いないので、スキャンに必要なコマンドは、`NOPASSAWORD`として、remote host上の`etc/sudoers`に定義しておく。 +See [Usage: Configtest#Check /etc/sudoers](#check-etcsudoers) + +## Step6. Check config.toml and settings on the server before scanning ``` $ vuls configtest ``` 詳細は [Usage: configtest](#usage-configtest) を参照 -## Step8. Start Scanning +## Step7. Start Scanning ``` @@ -234,11 +213,11 @@ $ vuls scan Scan Summary ============ -172-31-4-82 amazon 2015.09 94 CVEs 103 updatable packages +localhost amazon 2015.09 94 CVEs 103 updatable packages ``` -## Step9. Reporting +## Step8. Reporting View one-line summary @@ -247,7 +226,7 @@ $ vuls report -format-one-line-text -cvedb-path=$PWD/cve.sqlite3 One Line Summary ================ -172-31-4-82 Total: 94 (High:19 Medium:54 Low:7 ?:14) 103 updatable packages +localhost Total: 94 (High:19 Medium:54 Low:7 ?:14) 103 updatable packages ``` @@ -256,7 +235,7 @@ View short summary. ``` $ vuls report -format-short-text -cvedb-path=$PWD/cve.sqlite3 --lang=ja -172-31-4-8 (amazon 2015.09) +localhost (amazon 2015.09) =========================== Total: 94 (High:19 Medium:54 Low:7 ?:14) 103 updatable packages @@ -276,7 +255,7 @@ View full report. ``` $ vuls report -format-full-text -cvedb-path=$PWD/cve.sqlite3 --lang=ja -172-31-4-82 (amazon 2015.09) +localhost (amazon 2015.09) ============================ Total: 94 (High:19 Medium:54 Low:7 ?:14) 103 updatable packages @@ -307,7 +286,7 @@ Confidence 100 / YumUpdateSecurityMatch ... snip ... ``` -## Step10. TUI +## Step9. TUI Vulsにはスキャン結果の詳細を参照できるイカしたTUI(Terminal-Based User Interface)が付属している。 @@ -317,13 +296,105 @@ $ vuls tui ![Vuls-TUI](img/hello-vuls-tui.png) -## Step11. Web UI +## Step10. Web UI [VulsRepo](https://github.com/usiusi360/vulsrepo)はスキャン結果をビボットテーブルのように分析可能にするWeb UIである。 [Online Demo](http://usiusi360.github.io/vulsrepo/)があるので試してみて。 ---- +# Tutorial: Remote Scan Mode + +SSHを用いてリモートのホストをスキャンする方法を説明する。 + +1. Amazon Linuxを新規に1台作成(スキャン対象) +1. 必要なソフトウェアをインストール +1. RemoteホストにlocalhostからSSH可能にする +1. 設定 +1. 設定ファイルと、スキャン対象サーバの設定のチェック +1. Scan +1. Reporting + +先程のチュートリアルで作成したVulsサーバ(以下localhostと記述)を用いる。 + +## Step1. Launch Another Amazon Linux + +[Tutorial: Local Scan Mode#Step1. Launch Amazon Linux](#step1-launch-amazon-linux)と同じ +新規にターミナルを開いて今作成したEC2にSSH接続する。 + +## Step2. Install Dependencies on the Remote Server + +ディストリビューションによってはスキャンに必要な依存ソフトウェアをインストールする必要がある。 +これらはリモートサーバ上に手動かAnsibleなどでインストールする。 +依存ソフトウェアの詳細は [Dependencies on Target Servers](#dependencies-on-target-servers) を参照。 + +## Step3. Enable to SSH from Localhost + +VulsはSSHパスワード認証をサポートしてない。SSHの鍵認証の設定をしなければならない。 +localhost上でkeypairを作成し、remote host上のauthorized_keysに追加する。 + +- Localhost +```bash +$ ssh-keygen -t rsa +``` +Copy ~/.ssh/id_rsa.pub to the clipboard. + +- Remote Host +``` +$ mkdir ~/.ssh +$ chmod 700 ~/.ssh +$ touch ~/.ssh/authorized_keys +$ chmod 600 ~/.ssh/authorized_keys +$ vim ~/.ssh/authorized_keys +``` +Paste from the clipboard to ~/.ssh/.authorized_keys + +パスワードありのsudoはセキュリティ上の理由からサポート指定いないので、スキャンに必要なコマンドは、`NOPASSAWORD`として、remote host上の`etc/sudoers`に定義しておく。 +See [Usage: Configtest#Check /etc/sudoers](https://github.com/future-architect/vuls#check-etcsudoers) + +## Step4. Config + +- Localhost +``` +$ cd $HOME +$ cat config.toml +[servers] + +[servers.172-31-4-82] +host = "172.31.4.82" +port = "22" +user = "ec2-user" +keyPath = "/home/ec2-user/.ssh/id_rsa" +``` + +## Step5. Check config.toml and settings on the server before scanning + +``` +$ vuls configtest +``` + +see [Usage: configtest](#usage-configtest) + +## Step6. Start Scanning + +``` +$ vuls scan +... snip ... + +Scan Summary +============ +172-31-4-82 amazon 2015.09 94 CVEs 103 updatable packages + +``` + +## Step7. Reporting + +See [Tutorial: Local Scan Mode#Step8. Reporting](#step8-reporting) +See [Tutorial: Local Scan Mode#Step9. TUI](#step9-tui) +See [Tutorial: Local Scan Mode#Step10. Web UI](#step10-web-ui) + +---- + # Architecture ## A. Scan via SSH Mode (Remote Scan Mode) @@ -1242,7 +1313,7 @@ optional = [ ``` $ vuls report \ -cvedb-type=mysql \ - -cvedb-url="user:pass@tcp(localhost:3306)/dbname" + -cvedb-url="user:pass@tcp(localhost:3306)/dbname?parseTime=true" ``` ---- @@ -1303,33 +1374,39 @@ VulsとDependency Checkを連携すると以下の利点がある ## Display the latest scan results ``` -$ vuls tui -h tui: tui [-cvedb-type=sqlite3|mysql] [-cvedb-path=/path/to/cve.sqlite3] [-cvedb-url=http://127.0.0.1:1323 or mysql connection string] - [-results-dir=/path/to/results] [-refresh-cve] + [-results-dir=/path/to/results] + [-log-dir=/path/to/log] + [-debug] [-debug-sql] [-pipe] -cvedb-path string - /path/to/sqlite3 (For get cve detail from cve.sqlite3) + /path/to/sqlite3 (For get cve detail from cve.sqlite3) -cvedb-type string - DB type for fetching CVE dictionary (sqlite3 or mysql) + DB type for fetching CVE dictionary (sqlite3 or mysql) (default "sqlite3") -cvedb-url string http://cve-dictionary.com:8080 or mysql connection string + -debug + debug mode -debug-sql debug SQL + -log-dir string + /path/to/log (default "/var/log/vuls") -pipe Use stdin via PIPE -refresh-cve Refresh CVE information in JSON file under results dir -results-dir string - /path/to/results + /path/to/results ``` + Key binding is below. | key | | diff --git a/README.md b/README.md index eaae1c47..12863313 100644 --- a/README.md +++ b/README.md @@ -78,17 +78,16 @@ You can run install and run Vuls on your machine with only a few commands. see https://github.com/future-architect/vuls/tree/master/setup/docker - Manually -Hello Vuls Tutorial shows how to setup vuls manually. +Tutorial shows how to setup vuls manually. ---- -# Tutorial: Hello Vuls +# Tutorial: Local Scan Mode This tutorial will let you scan the vulnerabilities on the localhost with Vuls. This can be done in the following steps. 1. Launch Amazon Linux -1. Enable to ssh from localhost 1. Install requirements 1. Deploy go-cve-dictionary 1. Deploy Vuls @@ -111,21 +110,7 @@ This can be done in the following steps. - [Q: How do I disable the automatic installation of critical and important security updates on initial launch?](https://aws.amazon.com/amazon-linux-ami/faqs/?nc1=h_ls) -## Step2. SSH setting - -This is required to ssh to itself. - -Create a keypair then append public key to authorized_keys -```bash -$ ssh-keygen -t rsa -$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys -$ chmod 600 ~/.ssh/authorized_keys -``` - -Vuls doesn't support SSH password authentication. So you have to use SSH key-based authentication. -And also, SUDO with password is not supported for security reasons. So you have to define NOPASSWORD in /etc/sudoers on target servers. - -## Step3. Install requirements +## Step2. Install requirements Vuls requires the following packages. @@ -156,7 +141,7 @@ Set the OS environment variable to current shell $ source /etc/profile.d/goenv.sh ``` -## Step4. Deploy [go-cve-dictionary](https://github.com/kotakanbe/go-cve-dictionary) +## Step3. Deploy [go-cve-dictionary](https://github.com/kotakanbe/go-cve-dictionary) ```bash $ sudo mkdir /var/log/vuls @@ -182,7 +167,7 @@ $ ls -alh cve.sqlite3 -rw-r--r-- 1 ec2-user ec2-user 7.0M Mar 24 13:20 cve.sqlite3 ``` -## Step5. Deploy Vuls +## Step4. Deploy Vuls Launch a new terminal and SSH to the ec2 instance. @@ -195,25 +180,24 @@ $ make install ``` The binary was built under `$GOPATH/bin` -## Step6. Config +## Step5. Config Create a config file(TOML format). -Then check the config. - ``` $ cd $HOME $ cat config.toml [servers] -[servers.172-31-4-82] -host = "172.31.4.82" -port = "22" -user = "ec2-user" -keyPath = "/home/ec2-user/.ssh/id_rsa" - +[servers.localhost] +host = "localhost" +port = "local" ``` -## Step7. Check config.toml and settings on the server before scanning +Root privilege is needed on Some distributions. +Sudo with password is not supported for security reasons. So you have to define NOPASSWORD in /etc/sudoers. +See [Usage: Configtest#Check /etc/sudoers](#check-etcsudoers) + +## Step6. Check config.toml and settings on the server before scanning ``` $ vuls configtest @@ -221,7 +205,7 @@ $ vuls configtest see [Usage: configtest](#usage-configtest) -## Step8. Start Scanning +## Step7. Start Scanning ``` $ vuls scan @@ -229,11 +213,11 @@ $ vuls scan Scan Summary ============ -172-31-4-82 amazon 2015.09 94 CVEs 103 updatable packages +localhost amazon 2015.09 94 CVEs 103 updatable packages ``` -## Step9. Reporting +## Step8. Reporting View one-line summary @@ -242,7 +226,7 @@ $ vuls report -format-one-line-text -cvedb-path=$PWD/cve.sqlite3 One Line Summary ================ -172-31-4-82 Total: 94 (High:19 Medium:54 Low:7 ?:14) 103 updatable packages +localhost Total: 94 (High:19 Medium:54 Low:7 ?:14) 103 updatable packages ``` @@ -251,7 +235,7 @@ View short summary. ``` $ vuls report -format-short-text -172-31-4-8 (amazon 2015.09) +localhost (amazon 2015.09) =========================== Total: 94 (High:19 Medium:54 Low:7 ?:14) 103 updatable packages @@ -273,7 +257,7 @@ View full report. ``` $ vuls report -format-full-text -172-31-4-82 (amazon 2015.09) +localhost (amazon 2015.09) ============================ Total: 94 (High:19 Medium:54 Low:7 ?:14) 103 updatable packages @@ -300,7 +284,7 @@ Confidence 100 / YumUpdateSecurityMatch ... snip ... ``` -## Step10. TUI +## Step9. TUI Vuls has Terminal-Based User Interface to display the scan result. @@ -310,13 +294,106 @@ $ vuls tui ![Vuls-TUI](img/hello-vuls-tui.png) -## Step11. Web UI +## Step10. Web UI [VulsRepo](https://github.com/usiusi360/vulsrepo) is a awesome Web UI for Vuls. Check it out the [Online Demo](http://usiusi360.github.io/vulsrepo/). ---- +# Tutorial: Remote Scan Mode + +This tutorial will let you scan the vulnerabilities on the remote host via SSH with Vuls. +This can be done in the following steps. + +1. Launch Another Amazon Linux +1. Install Dependencies on the Remote Host +1. Enable to SSH from Localhost +1. Configuration +1. Check config.toml and settings on the server before scanning +1. Scan +1. Reporting + +We will use the Vuls server (called localhost) created in the previous tutorial. + +## Step1. Launch Another Amazon Linux + +Same as [Tutorial: Local Scan Mode#Step1. Launch Amazon Linux](#step1-launch-amazon-linux) +Launch a new terminal and SSH to the Remote Server. + +## Step2. Install Dependencies on the Remote Server + +Depending on the distribution you need to install dependent modules. +Install these dependencies manually or using Ansible etc. +For details of dependent libraries, see [Dependencies on Target Servers](#dependencies-on-target-servers) + +## Step3. Enable to SSH from Localhost + +Vuls doesn't support SSH password authentication. So you have to use SSH key-based authentication. +Create a keypair on the localhost then append public key to authorized_keys on the remote host. + +- Localhost +```bash +$ ssh-keygen -t rsa +``` +Copy ~/.ssh/id_rsa.pub to the clipboard. + +- Remote Host +``` +$ mkdir ~/.ssh +$ chmod 700 ~/.ssh +$ touch ~/.ssh/authorized_keys +$ chmod 600 ~/.ssh/authorized_keys +$ vim ~/.ssh/authorized_keys +``` +Paste from the clipboard to ~/.ssh/.authorized_keys + +And also, SUDO with password is not supported for security reasons. So you have to define NOPASSWORD in /etc/sudoers on target servers. +See [Usage: Configtest#Check /etc/sudoers](https://github.com/future-architect/vuls#check-etcsudoers) + +## Step4. Config + +- Localhost +``` +$ cd $HOME +$ cat config.toml +[servers] + +[servers.172-31-4-82] +host = "172.31.4.82" +port = "22" +user = "ec2-user" +keyPath = "/home/ec2-user/.ssh/id_rsa" +``` + +## Step5. Check config.toml and settings on the server before scanning + +``` +$ vuls configtest +``` + +see [Usage: configtest](#usage-configtest) + +## Step6. Start Scanning + +``` +$ vuls scan +... snip ... + +Scan Summary +============ +172-31-4-82 amazon 2015.09 94 CVEs 103 updatable packages + +``` + +## Step7. Reporting + +See [Tutorial: Local Scan Mode#Step8. Reporting](#step8-reporting) +See [Tutorial: Local Scan Mode#Step9. TUI](#step9-tui) +See [Tutorial: Local Scan Mode#Step10. Web UI](#step10-web-ui) + +---- + # Setup Vuls in a Docker Container see https://github.com/future-architect/vuls/tree/master/setup/docker @@ -343,7 +420,7 @@ On the aggregation server, you can refer to the scanning result of each scan tar ## Scanning Flow ![Vuls-Scan-Flow](img/vuls-scan-flow.png) - Scan vulnerabilities on the servers via SSH and collect a list of the CVE ID - - To scan Docker containers, Vuls connect via ssh to the Docker host and then `docker exec` to the containers. So, no need to run sshd daemon on the containers. + - To scan Docker containers, Vuls connects via SSH to the Docker host and then `docker exec` to the containers. So, no need to run sshd daemon on the containers. ---- # Performance Considerations @@ -617,7 +694,7 @@ configtest: ``` The configtest subcommand checks the following -- Whether vuls is able to connect via ssh to servers/containers defined in the config.toml +- Whether vuls is able to connect via SSH to servers/containers defined in the config.toml - Whether Dependent package is installed on the scan target server - Check /etc/sudoers @@ -728,7 +805,7 @@ Vuls supports different types of SSH. By Default, using a native Go implementation from crypto/ssh. This is useful in situations where you may not have access to traditional UNIX tools. -To use external ssh command, specify this option. +To use external SSH command, specify this option. This is useful If you want to use ProxyCommand or cipher algorithm of SSH that is not supported by native go implementation. Don't forget to add below line to /etc/sudoers on the target servers. (username: vuls) ``` @@ -1239,7 +1316,7 @@ optional = [ ``` $ vuls report \ -cvedb-type=mysql \ - -cvedb-url="user:pass@tcp(localhost:3306)/dbname" + -cvedb-url="user:pass@tcp(localhost:3306)/dbname?parseTime=true" ``` ---- @@ -1293,8 +1370,6 @@ How to integrate Vuls with OWASP Dependency Check ``` - - # Usage: TUI ## Display the latest scan results @@ -1305,25 +1380,31 @@ tui: [-cvedb-type=sqlite3|mysql] [-cvedb-path=/path/to/cve.sqlite3] [-cvedb-url=http://127.0.0.1:1323 or mysql connection string] - [-results-dir=/path/to/results] [-refresh-cve] + [-results-dir=/path/to/results] + [-log-dir=/path/to/log] + [-debug] [-debug-sql] [-pipe] -cvedb-path string - /path/to/sqlite3 (For get cve detail from cve.sqlite3) (default "/Users/kotakanbe/go/src/github.com/future-architect/vuls/cve.sqlite3") + /path/to/sqlite3 (For get cve detail from cve.sqlite3) -cvedb-type string DB type for fetching CVE dictionary (sqlite3 or mysql) (default "sqlite3") -cvedb-url string http://cve-dictionary.com:8080 or mysql connection string + -debug + debug mode -debug-sql debug SQL + -log-dir string + /path/to/log (default "/var/log/vuls") -pipe Use stdin via PIPE -refresh-cve Refresh CVE information in JSON file under results dir -results-dir string - /path/to/results (default "/Users/kotakanbe/go/src/github.com/future-architect/vuls/results") + /path/to/results ``` Key binding is below. @@ -1437,7 +1518,7 @@ Run with --debug, --sql-debug option. - Adjusting Open File Limit [Riak docs](http://docs.basho.com/riak/latest/ops/tuning/open-files-limit/) is awesome. -- Does Vuls accept ssh connections with fish-shell or old zsh as the login shell? +- Does Vuls accept SSH connections with fish-shell or old zsh as the login shell? No, Vuls needs a user on the server for bash login. see also [#8](/../../issues/8) - Windows diff --git a/setup/docker/README.md b/setup/docker/README.md index 9d2a4328..e8ee721f 100644 --- a/setup/docker/README.md +++ b/setup/docker/README.md @@ -85,7 +85,7 @@ vuls v0.1.xxx xxxx 1. fetch nvd (vuls/go-cve-dictionary) 1. configuration (vuls/vuls) -1. prepare (vuls/vuls) +1. configtest (vuls/vuls) 1. scan (vuls/vuls) 1. vulsrepo (vuls/vulsrepo) @@ -113,7 +113,7 @@ port = "22" user = "vuls-user" keyPath = "/root/.ssh/id_rsa" # path to ssh private key in docker ``` -  + ```console $ docker run --rm \ @@ -124,14 +124,14 @@ $ docker run --rm \ -config=./config.toml # path to config.toml in docker ``` -## Step3. Prepare +## Step3. Configtest ```console $ docker run --rm -it\ -v ~/.ssh:/root/.ssh:ro \ -v $PWD:/vuls \ -v $PWD/vuls-log:/var/log/vuls \ - vuls/vuls prepare \ + vuls/vuls configtest \ -config=./config.toml # path to config.toml in docker ``` diff --git a/setup/docker/vuls/latest/README.md b/setup/docker/vuls/latest/README.md index 0137ff3b..33ce9b38 100644 --- a/setup/docker/vuls/latest/README.md +++ b/setup/docker/vuls/latest/README.md @@ -39,7 +39,7 @@ Please see the [Documentation](https://github.com/future-architect/vuls) $ docker run --rm vuls/vuls -v ``` -## configtest +## config Create config.toml referring to [this](https://github.com/future-architect/vuls#configuration). @@ -52,25 +52,16 @@ port = "22" user = "vuls-user" keyPath = "/root/.ssh/id_rsa" # path to ssh private key in docker ``` -  - -```console -$ docker run --rm \ - -v ~/.ssh:/root/.ssh:ro \ - -v $PWD:/vuls \ - -v $PWD/vuls-log:/var/log/vuls \ - vuls/vuls configtest -``` -## prepare +## configtest ```console $ docker run --rm -it \ -v ~/.ssh:/root/.ssh:ro \ -v $PWD:/vuls \ -v $PWD/vuls-log:/var/log/vuls \ - vuls/vuls prepare \ + vuls/vuls configtest \ -config=./config.toml # path to config.toml in docker ```