Stage/vuls
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add README.md, README.ja.md, img, NOTICE

Browse Source
This commit is contained in:
kota kanbe
2016-03-29 10:55:37 +09:00
parent 96a71d75e1
commit d6897b8d41
15 changed files with 3265 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

611
README.md Normal file
View File

@@ -0,0 +1,611 @@
# Vuls: VULnerability Scanner
Vulnerability scanner for Linux, agentless, written in golang.
We have a slack team. [Join slack team](http://goo.gl/forms/xm5KFo35tu)
[![asciicast](https://asciinema.org/a/3y9zrf950agiko7klg8abvyck.png)](https://asciinema.org/a/3y9zrf950agiko7klg8abvyck)
![Vuls-slack](img/vuls-slack-en.png)
[README in Japanese](https://github.com/future-architect/vuls/blob/master/README.ja.md)
----
# Abstract
- 毎日のように発見される脆弱性の調査、ソフトウェアアップデート作業はシステム管理者にとって大変なタスクである
- サービス停止リスクを恐れてパッケージマネージャの自動アップデート機能を使わずに手動で行うケースも多いが、手動での運用には以下の問題がある
- NVDやJVNなどの脆弱性データベースの新着情報をウォッチするのが大変
- サーバにインストールされているソフトウェアは膨大であり、全てを人が把握するのは困難
- 特にサーバ台数が多い場合は、新たに発見された脆弱性が自分の管理するどのサーバに該当するのかの調査コストが大きく、また漏れる可能性がある
- 最新情報を見逃したら脆弱性が放置されたままになる
- Vulsはサーバに存在する脆弱性を自動スキャンし、詳細情報をレポートする
- システムに関係あるもののみ教えてくれる
- その脆弱性に該当するサーバを教えてくれる
- 自動スキャンのため脆弱性対策漏れを防ぐことができる
- CRONなどで定期実行、レポートすることで脆弱性放置を防ぐことできる
- Vulsはシステム管理者の日々の脆弱性対応を助け、システムをセキュアに保つために有用なツールとなることを目指している
![Vuls-Motivation](img/vuls-motivation.png)
----
# Main Features
- Linuxサーバに存在する脆弱性をスキャン
- Ubuntu, Debian, CentOS, Amazon Linux, RHELに対応
- クラウド、オンプレミス、Docker
- OSパッケージ管理対象外のミドルウェアをスキャン
- プログラミング言語のライブラリやフレームワーク、ミドルウェアの脆弱性スキャン
- CPEに登録されているソフトウェアが対象
- エージェントレスアーキテクチャ
- スキャン対象サーバにSSH接続可能なマシン1台にセットアップするだけで動作
- 複数のSSH認証方式をサポート
- SSH agent
- SSH public key authentication (with password, empty password)
- Password authentication
- 設定ファイルのテンプレート自動生成
- CIDRを指定してサーバを自動検出、設定ファイルのテンプレートを生成
- EmailやSlackで通知可能（日本語でのレポートも可能）
- 付属するTerminal-Based User Interfaceビューアでは、Vim風キーバインドでスキャン結果を参照可能
----
# Hello Vuls
Describe how to scan the vulnerabilites that is included on the localhost.
Procedure is as follows.
1. Launch Amazon Linux
1. Enable to ssh from localhost
1. Install requirements
1. Deploy go-cve-dictionary
1. Deploy Vuls
1. Configuration
1. Prepare
1. Scan
1. TUI(Terminal-Based User Interface)
## 1. Launch Amazon Linux
- For the purpose of explanation, using the old AMI (amzn-ami-hvm-2015.09.1.x86_64-gp2 - ami-383c1956)
- Instance size: t2.medium
- Unable to work on t2.small, micro, nano, Because 2.3GB memory is needed during data fetch from NVD for the first time.
- After the initial data fetch has been completed, It is able to run on t2.nano
- Add the following to the cloud-init, to prevent from automatically update at the first launch.
- [Q: How do I disable the automatic installation of critical and important security updates on initial launch?](https://aws.amazon.com/amazon-linux-ami/faqs/?nc1=h_ls)
```
#cloud-config
repo_upgrade: none
```
## 2. Enable to ssh from localhost
Create a keypair then append public key to authorized_keys
```bash
$ ssh-keygen -t rsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
```
## 3. Install requirements
Vuls requires the following packages.
- sqlite
- git
- gcc
- go v1.6
- https://golang.org/doc/install
```bash
$ ssh ec2-user@52.100.100.100 -i ~/.ssh/private.pem
$ sudo yum -y install sqlite git gcc
$ wget https://storage.googleapis.com/golang/go1.6.linux-amd64.tar.gz
$ sudo tar -C /usr/local -xzf go1.6.linux-amd64.tar.gz
$ mkdir $HOME/go
```
Put these lines into /etc/profile.d/goenv.sh
```bash
export GOROOT=/usr/local/go
export GOPATH=$HOME/go
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin
```
Set the OS environment variable to current shell
```bash
$ source /etc/profile.d/goenv.sh
```
## 4. Deploy go-cve-dictionary
go get
```bash
$ sudo mkdir /var/log/vuls
$ sudo chown ec2-user /var/log/vuls
$ sudo chmod 700 /var/log/vuls
$ go get github.com/kotakanbe/go-cve-dictionary
```
Start go-cve-dictionary as server mode.
For the first time, go-cve-dictionary fetches vulnerability data from NVD.
It takes about 10 minutes (on AWS).
```bash
$ go-cve-dictionary server
... Fetching ...
$ ls -alh cve.sqlite3
-rw-r--r-- 1 ec2-user ec2-user 7.0M Mar 24 13:20 cve.sqlite3
```
Now we has vulnerbility data, So start as server mode again.
```bash
$ go-cve-dictionary server
[Mar 24 15:21:55] INFO Opening DB. datafile: /home/ec2-user/cve.sqlite3
[Mar 24 15:21:55] INFO Migrating DB
[Mar 24 15:21:56] INFO Starting HTTP Sever...
[Mar 24 15:21:56] INFO Listening on 127.0.0.1:1323
```
## 5. Deploy vuls
Launch a new terminal, SSH to the ec2.
go get
```
$ go get github.com/future-architect/vuls
```
## 6. Config
Create a config file(TOML format).
```
$ cat config.toml
[servers]
[servers.172-31-4-82]
host = "172.31.4.82"
port = "22"
user = "ec2-user"
keyPath = "/home/ec2-user/.ssh/id_rsa"
```
## 7. Prepare
```
$ vuls prepare
```
## 8. Scan
```
$ vuls scan
INFO[0000] Begin scannig (config: /home/ec2-user/config.toml)
... snip ...
172-31-4-82 (amazon 2015.09)
============================
CVE-2016-0494 10.0 Unspecified vulnerability in the Java SE and Java SE Embedded components in Oracle
Java SE 6u105, 7u91, and 8u66 and Java SE Embedded 8u65 allows remote attackers to
affect confidentiality, integrity, and availability via unknown vectors related to
2D.
... snip ...
CVE-2016-0494
-------------
Score 10.0 (High)
Vector (AV:N/AC:L/Au:N/C:C/I:C/A:C)
Summary Unspecified vulnerability in the Java SE and Java SE Embedded components in Oracle Java SE 6u105,
7u91, and 8u66 and Java SE Embedded 8u65 allows remote attackers to affect confidentiality,
integrity, and availability via unknown vectors related to 2D.
NVD https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2016-0494
MITRE https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-0494
CVE Details http://www.cvedetails.com/cve/CVE-2016-0494
CVSS Claculator https://nvd.nist.gov/cvss/v2-calculator?name=CVE-2016-0494&vector=(AV:N/AC:L/Au:N/C:C/I:C/A:C)
RHEL-CVE https://access.redhat.com/security/cve/CVE-2016-0494
ALAS-2016-643 https://alas.aws.amazon.com/ALAS-2016-643.html
Package/CPE java-1.7.0-openjdk-1.7.0.91-2.6.2.2.63.amzn1 -> java-1.7.0-openjdk-1:1.7.0.95-2.6.4.0.65.amzn1
```
## 9. TUI
Vuls has Terminal-Based User Interface to see the latest scan.
```
$ vuls tui
```
![Vuls-TUI](img/hello-vuls-tui.png)
----
# Architecture
![Vuls-Architecture](img/vuls-architecture.png)
## go-cve-dictinary
- Fetch vulnerbility information from NVD, JVN(Japanese), then insert into SQLite.
## Vuls
- Scan vulnerabilities that exist on servers, to get a list of the CVE ID
- In order to get more information of the detected CVE, send HTTP request to go-cve-dictinary
- Send the report by Slack, Email
- System operator can see the latest report by terminal
----
# Usecase
## Scan all servers
![Vuls-Usecase1](img/vuls-usecase-elb-rails-rds-all.png)
## Scan a single server
web/app server in the same configuration under the load balancer
![Vuls-Usecase2](img/vuls-usecase-elb-rails-rds-single.png)
----
# Support OS
| Distribution| Release |
|:------------|-------------------:|
| Ubuntu | 12, 14, 16|
| Debian | 7, 8|
| RHEL | 4, 5, 6, 7|
| CentOS | 5, 6, 7|
| Amazon Linux| All |
----
# Usage: Automatic Server Discovery
Discovery subcommand discovers active servers which specifed in CIDR range, then print the template of config file(TOML format) to terminal.
```
$ vuls discover -help
discover:
discover 192.168.0.0/24
```
## Exapmle
```
$ vuls discover 172.31.4.0/24
# Create config.toml using below and then ./vuls --config=/path/to/config.toml
[slack]
hookURL = "https://hooks.slack.com/services/abc123/defghijklmnopqrstuvwxyz"
channel = "#channel-name"
#channel = "#{servername}"
iconEmoji = ":ghost:"
authUser = "username"
notifyUsers = ["@username"]
[mail]
smtpAddr = "smtp.gmail.com"
smtpPort = 465
user = "username"
password = "password"
from = "from@address.com"
to = ["to@address.com"]
cc = ["cc@address.com"]
subjectPrefix = "[vuls]"
[default]
#port = "22"
#user = "username"
#password = "password"
#keyPath = "/home/username/.ssh/id_rsa"
#keyPassword = "password"
[servers]
[servers.172-31-4-82]
host = "172.31.4.82"
#port = "22"
#user = "root"
#password = "password"
#keyPath = "/home/username/.ssh/id_rsa"
#keyPassword = "password"
#cpeNames = [
# "cpe:/a:rubyonrails:ruby_on_rails:4.2.1",
#]
```
You can customize your configuration using this template.
----
# Configuration
- Slack section
```
[slack]
hookURL = "https://hooks.slack.com/services/abc123/defghijklmnopqrstuvwxyz"
channel = "#channel-name"
#channel = "#{servername}"
iconEmoji = ":ghost:"
authUser = "username"
notifyUsers = ["@username"]
```
- hookURL : Incomming webhook's URL
- channel : channel name.
- Mail section
```
[mail]
smtpAddr = "smtp.gmail.com"
smtpPort = 465
user = "username"
password = "password"
from = "from@address.com"
to = ["to@address.com"]
cc = ["cc@address.com"]
subjectPrefix = "[vuls]"
```
- Defualt section
```
[default]
#port = "22"
#user = "username"
#password = "password"
#keyPath = "/home/username/.ssh/id_rsa"
#keyPassword = "password"
```
- servers section
```
[servers]
[servers.172-31-4-82]
host = "172.31.4.82"
#port = "22"
#user = "root"
#password = "password"
#keyPath = "/home/username/.ssh/id_rsa"
#keyPassword = "password"
#cpeNames = [
# "cpe:/a:rubyonrails:ruby_on_rails:4.2.1",
#]
```
----
# Usage: Prepare
Prepare subcommand installs required packages on each server.
| Distribution| Release | Requirements |
|:------------|-------------------:|:-------------|
| Ubuntu | 12, 14, 16| - |
| Debian | 7, 8| apptitude |
| CentOS | 5| yum-plugin-security, yum-changelog |
| CentOS | 6, 7| yum-plugin-security, yum-plugin-changelog |
| Amazon | All | - |
| RHEL | 4, 5, 6, 7 | - |
```
$ vuls prepare -help
prepare:
prepare [-config=/path/to/config.toml] [-debug]
-config string
/path/to/toml (default "/home/ec2-user/config.toml")
-debug
debug mode
-use-unattended-upgrades
[Depricated] For Ubuntu, install unattended-upgrades
```
----
# Usage: Scan
```
$ vuls scan -help
scan:
scan
[-lang=en|ja]
[-config=/path/to/config.toml]
[-dbpath=/path/to/vuls.sqlite3]
[-cve-dictionary-url=http://127.0.0.1:1323]
[-cvss-over=7]
[-report-slack]
[-report-mail]
[-http-proxy=http://192.168.0.1:8080]
[-debug]
[-debug-sql]
-config string
/path/to/toml (default "$HOME/config.toml")
-cve-dictionary-url string
http://CVE.Dictionary (default: %s) (default "http://127.0.0.1:1323")
-cvss-over float
-cvss-over=6.5 means reporting CVSS Score 6.5 and over (default: 0 (means report all))
-dbpath string
/path/to/sqlite3 (default "$HOME/vuls.sqlite3")
-debug
debug mode
-debug-sql
SQL debug mode
-http-proxy string
http://proxy-url:port (default: empty)
-lang string
[en|ja] (default "en")
-report-mail
Email report
-report-slack
Slack report
-use-unattended-upgrades
[Depricated] For Ubuntu. Scan by unattended-upgrades or not (use apt-get upgrade --dry-run by default)
-use-yum-plugin-security
[Depricated] For CentOS 5. Scan by yum-plugin-security or not (use yum check-update by default)
```
## example
Run go-cve-dictionary as server mdoe before scanning.
```
$ go-cve-dictionary server
```
### Scan all servers defined in config file
```
$ vuls scan --report-slack --report-mail --cvss-over=7
```
Examples of the above are the following meanings...
- Scan all servers defined in config file
- Send scan resulsts to slack, email
- Only Reporting CVEs that CVSS score over 7
- Print scan result to terminal
### Scan specified servers
```
$ vuls scan server1 server2
```
Examples of the above are the following meanings...
- Scan only 2 servers. (server1, server2)
- Print scan result to terminal
----
# Usage: Scan vulnerability of non-OS package
Vuls is possible to detect vulnerabilities something you compiled by yourself or library of language, framework that has been registered in the [CPE](https://nvd.nist.gov/cpe.cfm).
- How to search CPE name by software name
- [NVD: Search Common Platform Enumerations (CPE)](https://web.nvd.nist.gov/view/cpe/search)
**Check CPE Naming Format: 2.2**
- Configuration
If you want to detect the vulnerbility of Ruby on Rails v4.2.1, define as below.
```
[servers]
[servers.172-31-4-82]
host = "172.31.4.82"
user = "ec2-user"
keyPath = "/home/username/.ssh/id_rsa"
cpeNames = [
"cpe:/a:rubyonrails:ruby_on_rails:4.2.1",
]
```
# Usage: How to update NVD data.
```
$ go-cve-dictionary fetchnvd -h
fetchnvd:
fetchnvd
[-last2y]
[-dbpath=/path/to/cve.sqlite3]
[-debug]
[-debug-sql]
-dbpath string
/path/to/sqlite3 (default "/home/ec2-user/cve.sqlite3")
-debug
debug mode
-debug-sql
SQL debug mode
-last2y
Refresh NVD data in the last two years.
```
- Fetch data of the entire period
```
$ go-cve-dictionary fetchnvd -entire
```
- Fetch data last 2 years
```
$ go-cve-dictionary fetchnvd -last2y
```
----
# Misc
- HTTP Proxy Support
If your system is behind HTTP proxy, you have to specify --http-proxy option.
- How to Daemonize go-cve-dictionary
Use Systemd, Upstart or supervisord, daemontools...
- How to update vulnerbility data automatically.
Use job scheduler like Cron (with -last2y option).
- How to cross compile
```bash
$ cd /path/to/your/local-git-reporsitory/vuls
$ GOOS=linux GOARCH=amd64 go build -o vuls.amd64
```
- Logging
Log wrote to under /var/log/vuls/
- Debug
Run with --debug, --sql-debug option.
- Windows
Use Microsoft Baseline Secuirty Analyzer. [MBSA](https://technet.microsoft.com/en-us/security/cc184924.aspx)
----
# Data Source
- [NVD](https://nvd.nist.gov/)
- [JVN(Japanese)](http://jvndb.jvn.jp/apis/myjvn/)
# Authors
kotakanbe ([@kotakanbe](https://twitter.com/kotakanbe)) created vuls and [these fine people](https://github.com/future-architect/vuls/graphs/contributors) have contributed.
----
# Contribute
1. Fork it
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create new Pull Request
----
# Change Log
Please see [CHANGELOG](https://github.com/future-architect/vuls/blob/master/CHANGELOG.md).
----
# Licence
Please see [LICENSE](https://github.com/future-architect/vuls/blob/master/LICENSE).