Merge pull request #270 from future-architect/report-subcommand

Add report subcommand, change scan options. #239

.gitignore

@@ -2,7 +2,9 @@ vuls
 .vscode
 *.txt
 *.json
-*.sqlite3
+*.sqlite3*
+*.db
+tags
 .gitmodules
 coverage.out
 issues/
@@ -10,3 +12,4 @@ vendor/
 log/
 results/
 *config.toml
+!setup/docker/*

CHANGELOG.md

@@ -1,5 +1,122 @@
 # Change Log
 
+## [v0.1.7](https://github.com/future-architect/vuls/tree/v0.1.7) (2016-11-08)
+[Full Changelog](https://github.com/future-architect/vuls/compare/v0.1.6...v0.1.7)
+
+**Implemented enhancements:**
+
+- Enable to scan only docker container, without docker host [\#122](https://github.com/future-architect/vuls/issues/122)
+- Add -skip-broken option \[CentOS only\] \#245 [\#248](https://github.com/future-architect/vuls/pull/248) ([kotakanbe](https://github.com/kotakanbe))
+- Display unknown CVEs to TUI [\#244](https://github.com/future-architect/vuls/pull/244) ([kotakanbe](https://github.com/kotakanbe))
+- Add the XML output [\#240](https://github.com/future-architect/vuls/pull/240) ([gleentea](https://github.com/gleentea))
+- add '-ssh-external' option to prepare subcommand [\#234](https://github.com/future-architect/vuls/pull/234) ([mykstmhr](https://github.com/mykstmhr))
+- Integrate OWASP Dependency Check [\#232](https://github.com/future-architect/vuls/pull/232) ([kotakanbe](https://github.com/kotakanbe))
+- Add support for reading CVE data from MySQL. [\#225](https://github.com/future-architect/vuls/pull/225) ([oswell](https://github.com/oswell))
+- Remove base docker image, -v shows commit hash [\#223](https://github.com/future-architect/vuls/pull/223) ([sadayuki-matsuno](https://github.com/sadayuki-matsuno))
+- Support ignore CveIDs in config [\#222](https://github.com/future-architect/vuls/pull/222) ([kotakanbe](https://github.com/kotakanbe))
+- Confirm before installing dependencies on prepare [\#219](https://github.com/future-architect/vuls/pull/219) ([kotakanbe](https://github.com/kotakanbe))
+- Remove all.json [\#218](https://github.com/future-architect/vuls/pull/218) ([kotakanbe](https://github.com/kotakanbe))
+- Add GitHub issue template [\#217](https://github.com/future-architect/vuls/pull/217) ([kotakanbe](https://github.com/kotakanbe))
+- Improve makefile, -version shows git hash, fix README [\#216](https://github.com/future-architect/vuls/pull/216) ([kotakanbe](https://github.com/kotakanbe))
+- change e-mail package from gomail to net/smtp [\#211](https://github.com/future-architect/vuls/pull/211) ([sadayuki-matsuno](https://github.com/sadayuki-matsuno))
+- Add only-containers option to scan subcommand \#122 [\#190](https://github.com/future-architect/vuls/pull/190) ([kotakanbe](https://github.com/kotakanbe))
+- Fix -results-dir option of scan subcommand [\#185](https://github.com/future-architect/vuls/pull/185) ([kotakanbe](https://github.com/kotakanbe))
+- Show error when no scannable servers are detected. [\#177](https://github.com/future-architect/vuls/pull/177) ([kotakanbe](https://github.com/kotakanbe))
+- Add sudo check to prepare subcommand [\#176](https://github.com/future-architect/vuls/pull/176) ([kotakanbe](https://github.com/kotakanbe))
+- Supports yum --enablerepo option \(supports only base,updates for now\) [\#147](https://github.com/future-architect/vuls/pull/147) ([kotakanbe](https://github.com/kotakanbe))
+
+**Fixed bugs:**
+
+- Debian 8.6 \(jessie\) scan does not show vulnerable packages [\#235](https://github.com/future-architect/vuls/issues/235)
+- panic: runtime error: index out of range - ubuntu 16.04 + vuls history [\#180](https://github.com/future-architect/vuls/issues/180)
+- Moved golang.org/x/net/context to context [\#243](https://github.com/future-architect/vuls/pull/243) ([yoheimuta](https://github.com/yoheimuta))
+- Fix changelog cache bug on Ubuntu and Debian \#235 [\#238](https://github.com/future-architect/vuls/pull/238) ([kotakanbe](https://github.com/kotakanbe))
+- add '-ssh-external' option to prepare subcommand [\#234](https://github.com/future-architect/vuls/pull/234) ([mykstmhr](https://github.com/mykstmhr))
+- Fixed error for the latest version of gocui [\#231](https://github.com/future-architect/vuls/pull/231) ([ymd38](https://github.com/ymd38))
+- Handle the refactored gocui SetCurrentView method. [\#229](https://github.com/future-architect/vuls/pull/229) ([oswell](https://github.com/oswell))
+- Fix locale env var LANG to LANGUAGE [\#215](https://github.com/future-architect/vuls/pull/215) ([kotakanbe](https://github.com/kotakanbe))
+- Fixed bug with parsing update line on CentOS/RHEL [\#206](https://github.com/future-architect/vuls/pull/206) ([andyone](https://github.com/andyone))
+- Fix defer cache.DB.close [\#201](https://github.com/future-architect/vuls/pull/201) ([kotakanbe](https://github.com/kotakanbe))
+- Fix a help message of -report-azure-blob option [\#195](https://github.com/future-architect/vuls/pull/195) ([kotakanbe](https://github.com/kotakanbe))
+- Fix error handling in tui [\#193](https://github.com/future-architect/vuls/pull/193) ([kotakanbe](https://github.com/kotakanbe))
+- Fix not working changelog cache on Container [\#189](https://github.com/future-architect/vuls/pull/189) ([kotakanbe](https://github.com/kotakanbe))
+- Fix release version detection on FreeBSD [\#184](https://github.com/future-architect/vuls/pull/184) ([kotakanbe](https://github.com/kotakanbe))
+- Fix defer cahce.DB.close\(\) [\#183](https://github.com/future-architect/vuls/pull/183) ([kotakanbe](https://github.com/kotakanbe))
+- Fix a mode of files/dir \(report, log\) [\#182](https://github.com/future-architect/vuls/pull/182) ([kotakanbe](https://github.com/kotakanbe))
+- Fix a error when no json dirs are found under results \#180 [\#181](https://github.com/future-architect/vuls/pull/181) ([kotakanbe](https://github.com/kotakanbe))
+- ssh-external option of configtest is not working \#178 [\#179](https://github.com/future-architect/vuls/pull/179) ([kotakanbe](https://github.com/kotakanbe))
+
+**Closed issues:**
+
+- Recent changes to gobui cause build failures [\#228](https://github.com/future-architect/vuls/issues/228)
+- https://hub.docker.com/r/vuls/go-cve-dictionary/ is empty [\#208](https://github.com/future-architect/vuls/issues/208)
+- Not able to install gomail fails [\#202](https://github.com/future-architect/vuls/issues/202)
+- No results file created - vuls tui failed [\#199](https://github.com/future-architect/vuls/issues/199)
+- Wrong file permissions for results/\*.json in official Docker container [\#197](https://github.com/future-architect/vuls/issues/197)
+- Failed: Unknown OS Type [\#196](https://github.com/future-architect/vuls/issues/196)
+- Segmentation fault with configtest [\#192](https://github.com/future-architect/vuls/issues/192)
+- Failed to scan. err: No server defined. Check the configuration [\#187](https://github.com/future-architect/vuls/issues/187)
+- vuls configtest -ssh-external doesnt work [\#178](https://github.com/future-architect/vuls/issues/178)
+- apt-get update: time out [\#175](https://github.com/future-architect/vuls/issues/175)
+- scanning on Centos6, but vuls recognizes debian. [\#174](https://github.com/future-architect/vuls/issues/174)
+
+**Merged pull requests:**
+
+- Update README \#225 [\#242](https://github.com/future-architect/vuls/pull/242) ([kotakanbe](https://github.com/kotakanbe))
+- fix readme [\#241](https://github.com/future-architect/vuls/pull/241) ([sadayuki-matsuno](https://github.com/sadayuki-matsuno))
+- Fix README \#234 [\#237](https://github.com/future-architect/vuls/pull/237) ([kotakanbe](https://github.com/kotakanbe))
+- Update glide files [\#236](https://github.com/future-architect/vuls/pull/236) ([kotakanbe](https://github.com/kotakanbe))
+- fix README [\#226](https://github.com/future-architect/vuls/pull/226) ([usiusi360](https://github.com/usiusi360))
+- fix some misspelling. [\#221](https://github.com/future-architect/vuls/pull/221) ([ymomoi](https://github.com/ymomoi))
+- fix docker readme [\#214](https://github.com/future-architect/vuls/pull/214) ([sadayuki-matsuno](https://github.com/sadayuki-matsuno))
+- Fix ja document about typo [\#213](https://github.com/future-architect/vuls/pull/213) ([shokohara](https://github.com/shokohara))
+- fix readme [\#212](https://github.com/future-architect/vuls/pull/212) ([sadayuki-matsuno](https://github.com/sadayuki-matsuno))
+- fix README [\#207](https://github.com/future-architect/vuls/pull/207) ([sadayuki-matsuno](https://github.com/sadayuki-matsuno))
+- fix typo [\#204](https://github.com/future-architect/vuls/pull/204) ([usiusi360](https://github.com/usiusi360))
+- fix gitignore [\#191](https://github.com/future-architect/vuls/pull/191) ([sadayuki-matsuno](https://github.com/sadayuki-matsuno))
+- Update glide.lock [\#188](https://github.com/future-architect/vuls/pull/188) ([kotakanbe](https://github.com/kotakanbe))
+- Fix path in setup/docker/README [\#186](https://github.com/future-architect/vuls/pull/186) ([dladuke](https://github.com/dladuke))
+- Vuls and vulsrepo are now separated [\#163](https://github.com/future-architect/vuls/pull/163) ([hikachan](https://github.com/hikachan))
+
+## [v0.1.6](https://github.com/future-architect/vuls/tree/v0.1.6) (2016-09-12)
+[Full Changelog](https://github.com/future-architect/vuls/compare/v0.1.5...v0.1.6)
+
+**Implemented enhancements:**
+
+- High speed scan on Ubuntu/Debian [\#172](https://github.com/future-architect/vuls/pull/172) ([kotakanbe](https://github.com/kotakanbe))
+- Support CWE\(Common Weakness Enumeration\) [\#169](https://github.com/future-architect/vuls/pull/169) ([kotakanbe](https://github.com/kotakanbe))
+- Enable to scan without sudo on amazon linux [\#167](https://github.com/future-architect/vuls/pull/167) ([kotakanbe](https://github.com/kotakanbe))
+- Remove deprecated options -use-unattended-upgrades,-use-yum-plugin-security [\#161](https://github.com/future-architect/vuls/pull/161) ([kotakanbe](https://github.com/kotakanbe))
+- delete sqlite3 [\#152](https://github.com/future-architect/vuls/pull/152) ([sadayuki-matsuno](https://github.com/sadayuki-matsuno))
+
+**Fixed bugs:**
+
+- Failed to setup vuls docker [\#170](https://github.com/future-architect/vuls/issues/170)
+- yum check-update error occurred when no reboot after kernel updating [\#165](https://github.com/future-architect/vuls/issues/165)
+- error thrown from 'docker build .' [\#157](https://github.com/future-architect/vuls/issues/157)
+- CVE-ID is truncated to 4 digits [\#153](https://github.com/future-architect/vuls/issues/153)
+- 'yum update --changelog' stalled in 'vuls scan'. if ssh user is not 'root'. [\#150](https://github.com/future-architect/vuls/issues/150)
+- Panic on packet scan [\#131](https://github.com/future-architect/vuls/issues/131)
+- Update glide.lock \#170 [\#171](https://github.com/future-architect/vuls/pull/171) ([kotakanbe](https://github.com/kotakanbe))
+- Fix detecting a platform on Azure [\#168](https://github.com/future-architect/vuls/pull/168) ([kotakanbe](https://github.com/kotakanbe))
+- Fix parse error for yum check-update \#165 [\#166](https://github.com/future-architect/vuls/pull/166) ([kotakanbe](https://github.com/kotakanbe))
+- Fix bug: Vuls on Docker [\#159](https://github.com/future-architect/vuls/pull/159) ([tjinjin](https://github.com/tjinjin))
+- Fix CVE-ID is truncated to 4 digits [\#155](https://github.com/future-architect/vuls/pull/155) ([usiusi360](https://github.com/usiusi360))
+- Fix yum update --changelog stalled when non-root ssh user on CentOS \#150 [\#151](https://github.com/future-architect/vuls/pull/151) ([kotakanbe](https://github.com/kotakanbe))
+
+**Closed issues:**
+
+- Support su for root privilege escalation [\#44](https://github.com/future-architect/vuls/issues/44)
+- Support FreeBSD [\#34](https://github.com/future-architect/vuls/issues/34)
+
+**Merged pull requests:**
+
+- Change scripts for data fetching from jvn [\#164](https://github.com/future-architect/vuls/pull/164) ([kotakanbe](https://github.com/kotakanbe))
+- Fix: setup vulsrepo [\#162](https://github.com/future-architect/vuls/pull/162) ([tjinjin](https://github.com/tjinjin))
+- Fix-docker-vulsrepo-install [\#160](https://github.com/future-architect/vuls/pull/160) ([usiusi360](https://github.com/usiusi360))
+- Reduce regular expression compilation [\#158](https://github.com/future-architect/vuls/pull/158) ([itchyny](https://github.com/itchyny))
+- Add testcases for \#153 [\#156](https://github.com/future-architect/vuls/pull/156) ([kotakanbe](https://github.com/kotakanbe))
+
 ## [v0.1.5](https://github.com/future-architect/vuls/tree/v0.1.5) (2016-08-16)
 [Full Changelog](https://github.com/future-architect/vuls/compare/v0.1.4...v0.1.5)
 

ISSUE_TEMPLATE

@@ -0,0 +1,36 @@
+
+# Environment
+
+## Vuls
+
+Hash : ____
+
+To check the commit hash of HEAD
+$ vuls -v
+
+or
+$ cd $GOPATH/src/github.com/future-architect/vuls 
+$ git rev-parse --short HEAD 
+
+## OS
+- Target Server: Write here
+- Vuls Server: Write here
+
+## Go
+- Go version: here
+
+# Current Output
+
+Please re-run the command using ```-debug``` and provide the output below.
+
+# Addition Details
+
+Can you also please fill in each of the remaining sections.
+
+## Expected Behavior
+
+## Actual Behavior
+
+## Steps to reproduce the behaviour
+
+

Makefile

@@ -1,4 +1,9 @@
 .PHONY: \
+	glide \
+	deps \
+	update \
+	build \
+	install \
 	all \
 	vendor \
 	lint \
@@ -11,14 +16,29 @@
 	clean
 
 SRCS = $(shell git ls-files '*.go')
-PKGS = ./. ./db ./config ./models ./report ./cveapi ./scan ./util ./commands
+PKGS = ./. ./config ./models ./report ./cveapi ./scan ./util ./commands ./cache
+VERSION := $(shell git describe --tags --abbrev=0)
+REVISION := $(shell git rev-parse --short HEAD)
+LDFLAGS := -X 'main.version=$(VERSION)' \
+	-X 'main.revision=$(REVISION)'
+
+glide:
+	go get github.com/Masterminds/glide
+
+deps: glide
+	glide install
+
+update: glide
+	glide update
+
+build: main.go deps
+	go build -ldflags "$(LDFLAGS)" -o vuls $<
+
+install: main.go deps
+	go install -ldflags "$(LDFLAGS)"
 
 all: test
 
-#  vendor:
-#          @ go get -v github.com/mjibson/party
-#          party -d external -c -u
-
 lint:
 	@ go get -v github.com/golang/lint/golint
 	$(foreach file,$(SRCS),golint $(file) || exit;)
@@ -36,7 +56,7 @@ fmtcheck:
 pretest: lint vet fmtcheck
 
 test: pretest
-	$(foreach pkg,$(PKGS),go test -v $(pkg) || exit;)
+	$(foreach pkg,$(PKGS),go test -cover -v $(pkg) || exit;)
 
 unused :
 	$(foreach pkg,$(PKGS),unused $(pkg);)

README.fr.md

@@ -107,14 +107,14 @@ Vuls requiert l'installation des paquets suivants :
 - sqlite
 - git
 - gcc
-- go v1.6
+- go v1.7.1 or later
     - https://golang.org/doc/install
 
 ```bash
 $ ssh ec2-user@52.100.100.100  -i ~/.ssh/private.pem
 $ sudo yum -y install sqlite git gcc
-$ wget https://storage.googleapis.com/golang/go1.6.linux-amd64.tar.gz
-$ sudo tar -C /usr/local -xzf go1.6.linux-amd64.tar.gz
+$ wget https://storage.googleapis.com/golang/go1.7.1.linux-amd64.tar.gz
+$ sudo tar -C /usr/local -xzf go1.7.1.linux-amd64.tar.gz
 $ mkdir $HOME/go
 ```
 Ajoutez les lignes suivantes dans /etc/profile.d/goenv.sh
@@ -201,7 +201,7 @@ Summary         Unspecified vulnerability in the Java SE and Java SE Embedded co
 NVD             https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2016-0494
 MITRE           https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-0494
 CVE Details     http://www.cvedetails.com/cve/CVE-2016-0494
-CVSS Claculator https://nvd.nist.gov/cvss/v2-calculator?name=CVE-2016-0494&vector=(AV:N/AC:L/Au:N/C:C/I:C/A:C)
+CVSS Calculator https://nvd.nist.gov/cvss/v2-calculator?name=CVE-2016-0494&vector=(AV:N/AC:L/Au:N/C:C/I:C/A:C)
 RHEL-CVE        https://access.redhat.com/security/cve/CVE-2016-0494
 ALAS-2016-643   https://alas.aws.amazon.com/ALAS-2016-643.html
 Package/CPE     java-1.7.0-openjdk-1.7.0.91-2.6.2.2.63.amzn1 -> java-1.7.0-openjdk-1:1.7.0.95-2.6.4.0.65.amzn1

cache/bolt.go

@@ -0,0 +1,173 @@
+/* Vuls - Vulnerability Scanner
+Copyright (C) 2016  Future Architect, Inc. Japan.
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+package cache
+
+import (
+	"encoding/json"
+	"fmt"
+
+	"github.com/Sirupsen/logrus"
+	"github.com/boltdb/bolt"
+	"github.com/future-architect/vuls/util"
+)
+
+// Bolt holds a pointer of bolt.DB
+// boltdb is used to store a cache of Changelogs of Ubuntu/Debian
+type Bolt struct {
+	Path string
+	Log  *logrus.Entry
+	db   *bolt.DB
+}
+
+// SetupBolt opens a boltdb and creates a meta bucket if not exists.
+func SetupBolt(path string, l *logrus.Entry) error {
+	l.Infof("Open boltDB: %s", path)
+	db, err := bolt.Open(path, 0600, nil)
+	if err != nil {
+		return err
+	}
+
+	b := Bolt{
+		Path: path,
+		Log:  l,
+		db:   db,
+	}
+	if err = b.createBucketIfNotExists(metabucket); err != nil {
+		return err
+	}
+
+	DB = b
+	return nil
+}
+
+// Close a db.
+func (b Bolt) Close() error {
+	if b.db == nil {
+		return nil
+	}
+	return b.db.Close()
+}
+
+//  CreateBucketIfNotExists creates a buket that is specified by arg.
+func (b *Bolt) createBucketIfNotExists(name string) error {
+	return b.db.Update(func(tx *bolt.Tx) error {
+		_, err := tx.CreateBucketIfNotExists([]byte(name))
+		if err != nil {
+			return fmt.Errorf("Failed to create bucket: %s", err)
+		}
+		return nil
+	})
+}
+
+// GetMeta gets a Meta Information os the servername to boltdb.
+func (b Bolt) GetMeta(serverName string) (meta Meta, found bool, err error) {
+	err = b.db.View(func(tx *bolt.Tx) error {
+		bkt := tx.Bucket([]byte(metabucket))
+		v := bkt.Get([]byte(serverName))
+		if len(v) == 0 {
+			found = false
+			return nil
+		}
+		if e := json.Unmarshal(v, &meta); e != nil {
+			return e
+		}
+		found = true
+		return nil
+	})
+	return
+}
+
+// EnsureBuckets puts a Meta information and create a buket that holds changelogs.
+func (b Bolt) EnsureBuckets(meta Meta) error {
+	jsonBytes, err := json.Marshal(meta)
+	if err != nil {
+		return fmt.Errorf("Failed to marshal to JSON: %s", err)
+	}
+	return b.db.Update(func(tx *bolt.Tx) error {
+		b.Log.Debugf("Put to meta: %s", meta.Name)
+		bkt := tx.Bucket([]byte(metabucket))
+		if err := bkt.Put([]byte(meta.Name), jsonBytes); err != nil {
+			return err
+		}
+
+		// re-create a bucket (bucket name: servername)
+		bkt = tx.Bucket([]byte(meta.Name))
+		if bkt != nil {
+			b.Log.Debugf("Delete bucket: %s", meta.Name)
+			if err := tx.DeleteBucket([]byte(meta.Name)); err != nil {
+				return err
+			}
+			b.Log.Debugf("Bucket deleted: %s", meta.Name)
+		}
+		b.Log.Debugf("Create bucket: %s", meta.Name)
+		if _, err := tx.CreateBucket([]byte(meta.Name)); err != nil {
+			return err
+		}
+		b.Log.Debugf("Bucket created: %s", meta.Name)
+		return nil
+	})
+}
+
+// PrettyPrint is for debuging
+func (b Bolt) PrettyPrint(meta Meta) error {
+	return b.db.View(func(tx *bolt.Tx) error {
+		bkt := tx.Bucket([]byte(metabucket))
+		v := bkt.Get([]byte(meta.Name))
+		b.Log.Debugf("key:%s, value:%s", meta.Name, v)
+
+		bkt = tx.Bucket([]byte(meta.Name))
+		c := bkt.Cursor()
+		for k, v := c.First(); k != nil; k, v = c.Next() {
+			b.Log.Debugf("key:%s, len: %d, %s...",
+				k, len(v), util.Truncate(string(v), 30))
+		}
+		return nil
+	})
+}
+
+// GetChangelog get the changelgo of specified packName from the Bucket
+func (b Bolt) GetChangelog(servername, packName string) (changelog string, err error) {
+	err = b.db.View(func(tx *bolt.Tx) error {
+		bkt := tx.Bucket([]byte(servername))
+		if bkt == nil {
+			return fmt.Errorf("Faild to get Bucket: %s", servername)
+		}
+		v := bkt.Get([]byte(packName))
+		if v == nil {
+			changelog = ""
+			return nil
+		}
+		changelog = string(v)
+		return nil
+	})
+	return
+}
+
+// PutChangelog put the changelgo of specified packName into the Bucket
+func (b Bolt) PutChangelog(servername, packName, changelog string) error {
+	return b.db.Update(func(tx *bolt.Tx) error {
+		bkt := tx.Bucket([]byte(servername))
+		if bkt == nil {
+			return fmt.Errorf("Faild to get Bucket: %s", servername)
+		}
+		if err := bkt.Put([]byte(packName), []byte(changelog)); err != nil {
+			return err
+		}
+		return nil
+	})
+}

cache/bolt_test.go

@@ -0,0 +1,134 @@
+/* Vuls - Vulnerability Scanner
+Copyright (C) 2016  Future Architect, Inc. Japan.
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+package cache
+
+import (
+	"os"
+	"reflect"
+	"testing"
+
+	"github.com/Sirupsen/logrus"
+	"github.com/boltdb/bolt"
+	"github.com/future-architect/vuls/config"
+	"github.com/future-architect/vuls/models"
+)
+
+const path = "/tmp/vuls-test-cache-11111111.db"
+const servername = "server1"
+
+var meta = Meta{
+	Name: servername,
+	Distro: config.Distro{
+		Family:  "ubuntu",
+		Release: "16.04",
+	},
+	Packs: []models.PackageInfo{
+		{
+			Name:    "apt",
+			Version: "1",
+		},
+	},
+}
+
+func TestSetupBolt(t *testing.T) {
+	log := logrus.NewEntry(&logrus.Logger{})
+	err := SetupBolt(path, log)
+	if err != nil {
+		t.Errorf("Failed to setup bolt: %s", err)
+	}
+	defer os.Remove(path)
+
+	if err := DB.Close(); err != nil {
+		t.Errorf("Failed to close bolt: %s", err)
+	}
+
+	// check if meta bucket exists
+	db, err := bolt.Open(path, 0600, nil)
+	if err != nil {
+		t.Errorf("Failed to open bolt: %s", err)
+	}
+
+	db.View(func(tx *bolt.Tx) error {
+		bkt := tx.Bucket([]byte(metabucket))
+		if bkt == nil {
+			t.Errorf("Meta bucket nof found")
+		}
+		return nil
+	})
+
+}
+
+func TestEnsureBuckets(t *testing.T) {
+	log := logrus.NewEntry(&logrus.Logger{})
+	if err := SetupBolt(path, log); err != nil {
+		t.Errorf("Failed to setup bolt: %s", err)
+	}
+	if err := DB.EnsureBuckets(meta); err != nil {
+		t.Errorf("Failed to ensure buckets: %s", err)
+	}
+	defer os.Remove(path)
+
+	m, found, err := DB.GetMeta(servername)
+	if err != nil {
+		t.Errorf("Failed to get meta: %s", err)
+	}
+	if !found {
+		t.Errorf("Not Found in meta")
+	}
+	if !reflect.DeepEqual(meta, m) {
+		t.Errorf("expected %v, actual %v", meta, m)
+	}
+	if err := DB.Close(); err != nil {
+		t.Errorf("Failed to close bolt: %s", err)
+	}
+
+	db, err := bolt.Open(path, 0600, nil)
+	if err != nil {
+		t.Errorf("Failed to open bolt: %s", err)
+	}
+	db.View(func(tx *bolt.Tx) error {
+		bkt := tx.Bucket([]byte(servername))
+		if bkt == nil {
+			t.Errorf("Meta bucket nof found")
+		}
+		return nil
+	})
+}
+
+func TestPutGetChangelog(t *testing.T) {
+	clog := "changelog-text"
+	log := logrus.NewEntry(&logrus.Logger{})
+	if err := SetupBolt(path, log); err != nil {
+		t.Errorf("Failed to setup bolt: %s", err)
+	}
+	defer os.Remove(path)
+
+	if err := DB.EnsureBuckets(meta); err != nil {
+		t.Errorf("Failed to ensure buckets: %s", err)
+	}
+	if err := DB.PutChangelog(servername, "apt", clog); err != nil {
+		t.Errorf("Failed to put changelog: %s", err)
+	}
+	if actual, err := DB.GetChangelog(servername, "apt"); err != nil {
+		t.Errorf("Failed to get changelog: %s", err)
+	} else {
+		if actual != clog {
+			t.Errorf("changelog is not same. e: %s, a: %s", clog, actual)
+		}
+	}
+}

cache/db.go

@@ -0,0 +1,56 @@
+/* Vuls - Vulnerability Scanner
+Copyright (C) 2016  Future Architect, Inc. Japan.
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+package cache
+
+import (
+	"github.com/future-architect/vuls/config"
+	"github.com/future-architect/vuls/models"
+)
+
+// DB has a cache instance
+var DB Cache
+
+const metabucket = "changelog-meta"
+
+// Cache is a interface of cache
+type Cache interface {
+	Close() error
+	GetMeta(string) (Meta, bool, error)
+	EnsureBuckets(Meta) error
+	PrettyPrint(Meta) error
+	GetChangelog(string, string) (string, error)
+	PutChangelog(string, string, string) error
+}
+
+// Meta holds a server name, distro information of the scanned server and
+// package information that was collected at the last scan.
+type Meta struct {
+	Name   string
+	Distro config.Distro
+	Packs  []models.PackageInfo
+}
+
+// FindPack search a PackageInfo
+func (m Meta) FindPack(name string) (pack models.PackageInfo, found bool) {
+	for _, p := range m.Packs {
+		if name == p.Name {
+			return p, true
+		}
+	}
+	return pack, false
+}

commands/configtest.go

@@ -18,6 +18,7 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
 package commands
 
 import (
+	"context"
 	"flag"
 	"io/ioutil"
 	"os"
@@ -26,7 +27,6 @@ import (
 
 	"github.com/Sirupsen/logrus"
 	"github.com/google/subcommands"
-	"golang.org/x/net/context"
 
 	c "github.com/future-architect/vuls/config"
 	"github.com/future-architect/vuls/scan"
@@ -97,6 +97,7 @@ func (p *ConfigtestCmd) Execute(_ context.Context, f *flag.FlagSet, _ ...interfa
 	}
 
 	c.Conf.Debug = p.debug
+	c.Conf.SSHExternal = p.sshExternal
 
 	err = c.Load(p.configPath, keyPass)
 	if err != nil {
@@ -145,12 +146,15 @@ func (p *ConfigtestCmd) Execute(_ context.Context, f *flag.FlagSet, _ ...interfa
 	Log := util.NewCustomLogger(c.ServerInfo{})
 
 	Log.Info("Validating Config...")
-	if !c.Conf.Validate() {
+	if !c.Conf.ValidateOnConfigtest() {
 		return subcommands.ExitUsageError
 	}
 
 	Log.Info("Detecting Server/Contianer OS... ")
-	scan.InitServers(Log)
+	if err := scan.InitServers(Log); err != nil {
+		Log.Errorf("Failed to init servers: %s", err)
+		return subcommands.ExitFailure
+	}
 
 	Log.Info("Checking sudo configuration... ")
 	if err := scan.CheckIfSudoNoPasswd(Log); err != nil {

commands/discover.go

@@ -18,6 +18,7 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
 package commands
 
 import (
+	"context"
 	"flag"
 	"fmt"
 	"os"
@@ -25,7 +26,6 @@ import (
 	"text/template"
 
 	"github.com/google/subcommands"
-	"golang.org/x/net/context"
 
 	"github.com/Sirupsen/logrus"
 	ps "github.com/kotakanbe/go-pingscanner"
@@ -98,9 +98,9 @@ iconEmoji    = ":ghost:"
 authUser     = "username"
 notifyUsers  = ["@username"]
 
-[mail]
+[email]
 smtpAddr      = "smtp.gmail.com"
-smtpPort      = "465"
+smtpPort      = "587"
 user          = "username"
 password      = "password"
 from          = "from@address.com"
@@ -115,7 +115,9 @@ subjectPrefix = "[vuls]"
 #cpeNames = [
 #  "cpe:/a:rubyonrails:ruby_on_rails:4.2.1",
 #]
+#dependencyCheckXMLPath = "/tmp/dependency-check-report.xml"
 #containers = ["${running}"]
+#ignoreCves = ["CVE-2014-6271"]
 #optional = [
 #    ["key", "value"],
 #]
@@ -131,7 +133,9 @@ host         = "{{$ip}}"
 #cpeNames = [
 #  "cpe:/a:rubyonrails:ruby_on_rails:4.2.1",
 #]
+#dependencyCheckXMLPath = "/tmp/dependency-check-report.xml"
 #containers = ["${running}"]
+#ignoreCves = ["CVE-2014-0160"]
 #optional = [
 #    ["key", "value"],
 #]

commands/history.go

@@ -18,27 +18,23 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
 package commands
 
 import (
+	"context"
 	"flag"
 	"fmt"
+	"io/ioutil"
 	"os"
 	"path/filepath"
 	"strings"
 
-	"golang.org/x/net/context"
-
-	"github.com/Sirupsen/logrus"
 	c "github.com/future-architect/vuls/config"
-	"github.com/future-architect/vuls/db"
-	"github.com/future-architect/vuls/models"
 	"github.com/google/subcommands"
 )
 
 // HistoryCmd is Subcommand of list scanned results
 type HistoryCmd struct {
-	debug    bool
-	debugSQL bool
-
-	dbpath string
+	debug      bool
+	debugSQL   bool
+	resultsDir string
 }
 
 // Name return subcommand name
@@ -53,7 +49,7 @@ func (*HistoryCmd) Synopsis() string {
 func (*HistoryCmd) Usage() string {
 	return `history:
 	history
-		[-dbpath=/path/to/vuls.sqlite3]
+		[-results-dir=/path/to/results]
 	`
 }
 
@@ -62,47 +58,41 @@ func (p *HistoryCmd) SetFlags(f *flag.FlagSet) {
 	f.BoolVar(&p.debugSQL, "debug-sql", false, "SQL debug mode")
 
 	wd, _ := os.Getwd()
-	defaultDBPath := filepath.Join(wd, "vuls.sqlite3")
-	f.StringVar(&p.dbpath, "dbpath", defaultDBPath, "/path/to/sqlite3")
+	defaultResultsDir := filepath.Join(wd, "results")
+	f.StringVar(&p.resultsDir, "results-dir", defaultResultsDir, "/path/to/results")
 }
 
 // Execute execute
 func (p *HistoryCmd) Execute(_ context.Context, f *flag.FlagSet, _ ...interface{}) subcommands.ExitStatus {
 
 	c.Conf.DebugSQL = p.debugSQL
-	c.Conf.DBPath = p.dbpath
+	c.Conf.ResultsDir = p.resultsDir
 
-	//  _, err := scanHistories()
-	histories, err := scanHistories()
-	if err != nil {
-		logrus.Error("Failed to select scan histories: ", err)
+	var err error
+	var dirs jsonDirs
+	if dirs, err = lsValidJSONDirs(); err != nil {
 		return subcommands.ExitFailure
 	}
-	const timeLayout = "2006-01-02 15:04"
-	for _, history := range histories {
-		names := []string{}
-		for _, result := range history.ScanResults {
-			if 0 < len(result.Container.ContainerID) {
-				names = append(names, result.Container.Name)
-			} else {
-				names = append(names, result.ServerName)
-			}
+	for _, d := range dirs {
+		var files []os.FileInfo
+		if files, err = ioutil.ReadDir(d); err != nil {
+			return subcommands.ExitFailure
 		}
-		fmt.Printf("%-3d %s scanned %d servers: %s\n",
-			history.ID,
-			history.ScannedAt.Format(timeLayout),
-			len(history.ScanResults),
-			strings.Join(names, ", "),
+		var hosts []string
+		for _, f := range files {
+			if filepath.Ext(f.Name()) != ".json" {
+				continue
+			}
+			fileBase := strings.TrimSuffix(f.Name(), filepath.Ext(f.Name()))
+			hosts = append(hosts, fileBase)
+		}
+		splitPath := strings.Split(d, string(os.PathSeparator))
+		timeStr := splitPath[len(splitPath)-1]
+		fmt.Printf("%s %d servers: %s\n",
+			timeStr,
+			len(hosts),
+			strings.Join(hosts, ", "),
 		)
 	}
 	return subcommands.ExitSuccess
 }
-
-func scanHistories() (histories []models.ScanHistory, err error) {
-	if err := db.OpenDB(); err != nil {
-		return histories, fmt.Errorf(
-			"Failed to open DB. datafile: %s, err: %s", c.Conf.DBPath, err)
-	}
-	histories, err = db.SelectScanHistories()
-	return
-}

commands/prepare.go

@@ -18,6 +18,7 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
 package commands
 
 import (
+	"context"
 	"flag"
 	"os"
 	"path/filepath"
@@ -27,7 +28,6 @@ import (
 	"github.com/future-architect/vuls/scan"
 	"github.com/future-architect/vuls/util"
 	"github.com/google/subcommands"
-	"golang.org/x/net/context"
 )
 
 // PrepareCmd is Subcommand of host discovery mode
@@ -38,7 +38,8 @@ type PrepareCmd struct {
 	askSudoPassword bool
 	askKeyPassword  bool
 
-	useUnattendedUpgrades bool
+	sshExternal bool
+	assumeYes   bool
 }
 
 // Name return subcommand name
@@ -46,12 +47,12 @@ func (*PrepareCmd) Name() string { return "prepare" }
 
 // Synopsis return synopsis
 func (*PrepareCmd) Synopsis() string {
-	//  return "Install packages Ubuntu: unattended-upgrade, CentOS: yum-plugin-security)"
 	return `Install required packages to scan.
 				CentOS: yum-plugin-security, yum-plugin-changelog
 				Amazon: None
-				RHEL:   TODO
+				RHEL:   None
 				Ubuntu: None
+				Debian: aptitude
 
 	`
 }
@@ -62,9 +63,11 @@ func (*PrepareCmd) Usage() string {
 	prepare
 			[-config=/path/to/config.toml]
 			[-ask-key-password]
+			[-assume-yes]
 			[-debug]
+			[-ssh-external]
 
-		    [SERVER]...
+			[SERVER]...
 `
 }
 
@@ -89,15 +92,21 @@ func (p *PrepareCmd) SetFlags(f *flag.FlagSet) {
 		&p.askSudoPassword,
 		"ask-sudo-password",
 		false,
-		"[Deprecated] THIS OPTION WAS REMOVED FOR SECURITY REASON. Define NOPASSWD in /etc/sudoers on tareget servers and use SSH key-based authentication",
+		"[Deprecated] THIS OPTION WAS REMOVED FOR SECURITY REASONS. Define NOPASSWD in /etc/sudoers on target servers and use SSH key-based authentication",
 	)
 
 	f.BoolVar(
-		&p.useUnattendedUpgrades,
-		"use-unattended-upgrades",
+		&p.sshExternal,
+		"ssh-external",
 		false,
-		"[Deprecated] For Ubuntu, install unattended-upgrades",
-	)
+		"Use external ssh command. Default: Use the Go native implementation")
+
+	f.BoolVar(
+		&p.assumeYes,
+		"assume-yes",
+		false,
+		"Assume any dependencies should be installed")
+
 }
 
 // Execute execute
@@ -112,7 +121,7 @@ func (p *PrepareCmd) Execute(_ context.Context, f *flag.FlagSet, _ ...interface{
 		}
 	}
 	if p.askSudoPassword {
-		logrus.Errorf("[Deprecated] -ask-sudo-password WAS REMOVED FOR SECURITY REASONS. Define NOPASSWD in /etc/sudoers on tareget servers and use SSH key-based authentication")
+		logrus.Errorf("[Deprecated] -ask-sudo-password WAS REMOVED FOR SECURITY REASONS. Define NOPASSWD in /etc/sudoers on target servers and use SSH key-based authentication")
 		return subcommands.ExitFailure
 	}
 
@@ -143,22 +152,34 @@ func (p *PrepareCmd) Execute(_ context.Context, f *flag.FlagSet, _ ...interface{
 	}
 
 	c.Conf.Debug = p.debug
-	c.Conf.UseUnattendedUpgrades = p.useUnattendedUpgrades
+	c.Conf.SSHExternal = p.sshExternal
+	c.Conf.AssumeYes = p.assumeYes
 
+	logrus.Info("Validating Config...")
+	if !c.Conf.ValidateOnPrepare() {
+		return subcommands.ExitUsageError
+	}
 	// Set up custom logger
 	logger := util.NewCustomLogger(c.ServerInfo{})
 
 	logger.Info("Detecting OS... ")
-	scan.InitServers(logger)
+	if err := scan.InitServers(logger); err != nil {
+		logger.Errorf("Failed to init servers: %s", err)
+		return subcommands.ExitFailure
+	}
+
+	logger.Info("Checking sudo configuration... ")
+	if err := scan.CheckIfSudoNoPasswd(logger); err != nil {
+		logger.Errorf("Failed to sudo with nopassword via SSH. Define NOPASSWD in /etc/sudoers on target servers")
+		return subcommands.ExitFailure
+	}
 
-	logger.Info("Installing...")
 	if errs := scan.Prepare(); 0 < len(errs) {
 		for _, e := range errs {
-			logger.Errorf("Failed: %s", e)
+			logger.Errorf("Failed to prepare: %s", e)
 		}
 		return subcommands.ExitFailure
 	}
 
-	logger.Info("Success")
 	return subcommands.ExitSuccess
 }

commands/report.go

@@ -0,0 +1,386 @@
+/* Vuls - Vulnerability Scanner
+Copyright (C) 2016  Future Architect, Inc. Japan.
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+package commands
+
+import (
+	"context"
+	"flag"
+	"fmt"
+	"os"
+	"path/filepath"
+
+	c "github.com/future-architect/vuls/config"
+	"github.com/future-architect/vuls/cveapi"
+	"github.com/future-architect/vuls/models"
+	"github.com/future-architect/vuls/report"
+	"github.com/future-architect/vuls/util"
+	"github.com/google/subcommands"
+	"github.com/kotakanbe/go-cve-dictionary/log"
+)
+
+// ReportCmd is subcommand for reporting
+type ReportCmd struct {
+	lang       string
+	debug      bool
+	debugSQL   bool
+	configPath string
+	resultsDir string
+	refreshCve bool
+
+	cvssScoreOver      float64
+	ignoreUnscoredCves bool
+	httpProxy          string
+
+	cvedbtype        string
+	cvedbpath        string
+	cveDictionaryURL string
+
+	toSlack     bool
+	toEMail     bool
+	toLocalFile bool
+	toS3        bool
+	toAzureBlob bool
+
+	formatJSON        bool
+	formatXML         bool
+	formatOneLineText bool
+	formatShortText   bool
+	formatFullText    bool
+
+	gzip bool
+
+	awsProfile  string
+	awsS3Bucket string
+	awsRegion   string
+
+	azureAccount   string
+	azureKey       string
+	azureContainer string
+}
+
+// Name return subcommand name
+func (*ReportCmd) Name() string { return "report" }
+
+// Synopsis return synopsis
+func (*ReportCmd) Synopsis() string { return "Reporting" }
+
+// Usage return usage
+func (*ReportCmd) Usage() string {
+	return `report:
+	report
+		[-lang=en|ja]
+		[-config=/path/to/config.toml]
+		[-results-dir=/path/to/results]
+		[-refresh-cve]
+		[-cvedb-type=sqlite3|mysql]
+		[-cvedb-path=/path/to/cve.sqlite3]
+		[-cvedb-url=http://127.0.0.1:1323 or mysql connection string]
+		[-cvss-over=7]
+		[-ignore-unscored-cves]
+		[-to-email]
+		[-to-slack]
+		[-to-localfile]
+		[-to-s3]
+		[-to-azure-blob]
+		[-format-json]
+		[-format-xml]
+		[-format-one-line-text]
+		[-format-short-text]
+		[-format-full-text]
+		[-gzip]
+		[-aws-profile=default]
+		[-aws-region=us-west-2]
+		[-aws-s3-bucket=bucket_name]
+		[-azure-account=accout]
+		[-azure-key=key]
+		[-azure-container=container]
+		[-http-proxy=http://192.168.0.1:8080]
+		[-debug]
+		[-debug-sql]
+
+		[SERVER]...
+`
+}
+
+// SetFlags set flag
+func (p *ReportCmd) SetFlags(f *flag.FlagSet) {
+	f.StringVar(&p.lang, "lang", "en", "[en|ja]")
+	f.BoolVar(&p.debug, "debug", false, "debug mode")
+	f.BoolVar(&p.debugSQL, "debug-sql", false, "SQL debug mode")
+
+	wd, _ := os.Getwd()
+
+	defaultConfPath := filepath.Join(wd, "config.toml")
+	f.StringVar(&p.configPath, "config", defaultConfPath, "/path/to/toml")
+
+	defaultResultsDir := filepath.Join(wd, "results")
+	f.StringVar(&p.resultsDir, "results-dir", defaultResultsDir, "/path/to/results")
+
+	f.BoolVar(
+		&p.refreshCve,
+		"refresh-cve",
+		false,
+		"Refresh CVE information in JSON file under results dir")
+
+	f.StringVar(
+		&p.cvedbtype,
+		"cvedb-type",
+		"sqlite3",
+		"DB type for fetching CVE dictionary (sqlite3 or mysql)")
+
+	defaultCveDBPath := filepath.Join(wd, "cve.sqlite3")
+	f.StringVar(
+		&p.cvedbpath,
+		"cvedb-path",
+		defaultCveDBPath,
+		"/path/to/sqlite3 (For get cve detail from cve.sqlite3)")
+
+	f.StringVar(
+		&p.cveDictionaryURL,
+		"cvedb-url",
+		"",
+		"http://cve-dictionary.com:8080 or mysql connection string")
+
+	f.Float64Var(
+		&p.cvssScoreOver,
+		"cvss-over",
+		0,
+		"-cvss-over=6.5 means reporting CVSS Score 6.5 and over (default: 0 (means report all))")
+
+	f.BoolVar(
+		&p.ignoreUnscoredCves,
+		"ignore-unscored-cves",
+		false,
+		"Don't report the unscored CVEs")
+
+	f.StringVar(
+		&p.httpProxy,
+		"http-proxy",
+		"",
+		"http://proxy-url:port (default: empty)")
+
+	f.BoolVar(&p.formatJSON,
+		"format-json",
+		false,
+		fmt.Sprintf("JSON format"))
+
+	f.BoolVar(&p.formatXML,
+		"format-xml",
+		false,
+		fmt.Sprintf("XML format"))
+
+	f.BoolVar(&p.formatOneLineText,
+		"format-one-line-text",
+		false,
+		fmt.Sprintf("One line summary in plain text"))
+
+	f.BoolVar(&p.formatShortText,
+		"format-short-text",
+		false,
+		fmt.Sprintf("Summary in plain text"))
+
+	f.BoolVar(&p.formatFullText,
+		"format-full-text",
+		false,
+		fmt.Sprintf("Detail report in plain text"))
+
+	f.BoolVar(&p.gzip, "gzip", false, "gzip compression")
+
+	f.BoolVar(&p.toSlack, "to-slack", false, "Send report via Slack")
+	f.BoolVar(&p.toEMail, "to-email", false, "Send report via Email")
+	f.BoolVar(&p.toLocalFile,
+		"to-localfile",
+		false,
+		fmt.Sprintf("Write report to localfile"))
+
+	f.BoolVar(&p.toS3,
+		"to-s3",
+		false,
+		"Write report to S3 (bucket/yyyyMMdd_HHmm/servername.json/xml/txt)")
+	f.StringVar(&p.awsProfile, "aws-profile", "default", "AWS profile to use")
+	f.StringVar(&p.awsRegion, "aws-region", "us-east-1", "AWS region to use")
+	f.StringVar(&p.awsS3Bucket, "aws-s3-bucket", "", "S3 bucket name")
+
+	f.BoolVar(&p.toAzureBlob,
+		"to-azure-blob",
+		false,
+		"Write report to Azure Storage blob (container/yyyyMMdd_HHmm/servername.json/xml/txt)")
+	f.StringVar(&p.azureAccount,
+		"azure-account",
+		"",
+		"Azure account name to use. AZURE_STORAGE_ACCOUNT environment variable is used if not specified")
+	f.StringVar(&p.azureKey,
+		"azure-key",
+		"",
+		"Azure account key to use. AZURE_STORAGE_ACCESS_KEY environment variable is used if not specified")
+	f.StringVar(&p.azureContainer, "azure-container", "", "Azure storage container name")
+}
+
+// Execute execute
+func (p *ReportCmd) Execute(_ context.Context, f *flag.FlagSet, _ ...interface{}) subcommands.ExitStatus {
+	c.Conf.Debug = p.debug
+	c.Conf.DebugSQL = p.debugSQL
+	Log := util.NewCustomLogger(c.ServerInfo{})
+
+	if err := c.Load(p.configPath, ""); err != nil {
+		Log.Errorf("Error loading %s, %s", p.configPath, err)
+		return subcommands.ExitUsageError
+	}
+
+	c.Conf.Lang = p.lang
+	c.Conf.ResultsDir = p.resultsDir
+	c.Conf.CveDBType = p.cvedbtype
+	c.Conf.CveDBPath = p.cvedbpath
+	c.Conf.CveDictionaryURL = p.cveDictionaryURL
+	c.Conf.CvssScoreOver = p.cvssScoreOver
+	c.Conf.IgnoreUnscoredCves = p.ignoreUnscoredCves
+	c.Conf.HTTPProxy = p.httpProxy
+
+	jsonDir, err := jsonDir(f.Args())
+	if err != nil {
+		log.Errorf("Failed to read from JSON: %s", err)
+		return subcommands.ExitFailure
+	}
+
+	c.Conf.FormatXML = p.formatXML
+	c.Conf.FormatJSON = p.formatJSON
+	c.Conf.FormatOneLineText = p.formatOneLineText
+	c.Conf.FormatShortText = p.formatShortText
+	c.Conf.FormatFullText = p.formatFullText
+
+	c.Conf.GZIP = p.gzip
+
+	// report
+	reports := []report.ResultWriter{
+		report.StdoutWriter{},
+	}
+
+	if p.toSlack {
+		reports = append(reports, report.SlackWriter{})
+	}
+
+	if p.toEMail {
+		reports = append(reports, report.EMailWriter{})
+	}
+
+	if p.toLocalFile {
+		reports = append(reports, report.LocalFileWriter{
+			CurrentDir: jsonDir,
+		})
+	}
+
+	if p.toS3 {
+		c.Conf.AwsRegion = p.awsRegion
+		c.Conf.AwsProfile = p.awsProfile
+		c.Conf.S3Bucket = p.awsS3Bucket
+		if err := report.CheckIfBucketExists(); err != nil {
+			Log.Errorf("Check if there is a bucket beforehand: %s, err: %s", c.Conf.S3Bucket, err)
+			return subcommands.ExitUsageError
+		}
+		reports = append(reports, report.S3Writer{})
+	}
+
+	if p.toAzureBlob {
+		c.Conf.AzureAccount = p.azureAccount
+		if len(c.Conf.AzureAccount) == 0 {
+			c.Conf.AzureAccount = os.Getenv("AZURE_STORAGE_ACCOUNT")
+		}
+
+		c.Conf.AzureKey = p.azureKey
+		if len(c.Conf.AzureKey) == 0 {
+			c.Conf.AzureKey = os.Getenv("AZURE_STORAGE_ACCESS_KEY")
+		}
+
+		c.Conf.AzureContainer = p.azureContainer
+		if len(c.Conf.AzureContainer) == 0 {
+			Log.Error("Azure storage container name is requied with --azure-container option")
+			return subcommands.ExitUsageError
+		}
+		if err := report.CheckIfAzureContainerExists(); err != nil {
+			Log.Errorf("Check if there is a container beforehand: %s, err: %s", c.Conf.AzureContainer, err)
+			return subcommands.ExitUsageError
+		}
+		reports = append(reports, report.AzureBlobWriter{})
+	}
+
+	if !(p.formatJSON || p.formatOneLineText ||
+		p.formatShortText || p.formatFullText || p.formatXML) {
+		c.Conf.FormatShortText = true
+	}
+
+	Log.Info("Validating Config...")
+	if !c.Conf.ValidateOnReport() {
+		return subcommands.ExitUsageError
+	}
+	if ok, err := cveapi.CveClient.CheckHealth(); !ok {
+		Log.Errorf("CVE HTTP server is not running. err: %s", err)
+		Log.Errorf("Run go-cve-dictionary as server mode before reporting or run with --cvedb-path option")
+		return subcommands.ExitFailure
+	}
+	if c.Conf.CveDictionaryURL != "" {
+		Log.Infof("cve-dictionary: %s", c.Conf.CveDictionaryURL)
+	} else {
+		if c.Conf.CveDBType == "sqlite3" {
+			Log.Infof("cve-dictionary: %s", c.Conf.CveDBPath)
+		}
+	}
+
+	history, err := loadOneScanHistory(jsonDir)
+
+	var results []models.ScanResult
+	for _, r := range history.ScanResults {
+		if p.refreshCve || needToRefreshCve(r) {
+			Log.Debugf("need to refresh")
+			if c.Conf.CveDBType == "sqlite3" {
+				if _, err := os.Stat(c.Conf.CveDBPath); os.IsNotExist(err) {
+					log.Errorf("SQLite3 DB(CVE-Dictionary) is not exist: %s",
+						c.Conf.CveDBPath)
+					return subcommands.ExitFailure
+				}
+			}
+
+			filled, err := fillCveInfoFromCveDB(r)
+			if err != nil {
+				Log.Errorf("Failed to fill CVE information: %s", err)
+				return subcommands.ExitFailure
+			}
+			filled.Lang = c.Conf.Lang
+
+			if err := overwriteJSONFile(jsonDir, filled); err != nil {
+				Log.Errorf("Failed to write JSON: %s", err)
+				return subcommands.ExitFailure
+			}
+			results = append(results, filled)
+		} else {
+			Log.Debugf("no need to refresh")
+			results = append(results, r)
+		}
+	}
+
+	var res models.ScanResults
+	for _, r := range results {
+		res = append(res, r.FilterByCvssOver())
+	}
+	for _, w := range reports {
+		if err := w.Write(res...); err != nil {
+			Log.Errorf("Failed to report: %s", err)
+			return subcommands.ExitFailure
+		}
+	}
+	return subcommands.ExitSuccess
+}

commands/scan.go

@@ -18,6 +18,7 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
 package commands
 
 import (
+	"context"
 	"flag"
 	"fmt"
 	"io/ioutil"
@@ -27,54 +28,23 @@ import (
 
 	"github.com/Sirupsen/logrus"
 	c "github.com/future-architect/vuls/config"
-	"github.com/future-architect/vuls/cveapi"
-	"github.com/future-architect/vuls/db"
-	"github.com/future-architect/vuls/report"
 	"github.com/future-architect/vuls/scan"
 	"github.com/future-architect/vuls/util"
 	"github.com/google/subcommands"
-	"golang.org/x/net/context"
+	"github.com/k0kubun/pp"
 )
 
 // ScanCmd is Subcommand of host discovery mode
 type ScanCmd struct {
-	lang     string
-	debug    bool
-	debugSQL bool
-
-	configPath string
-
-	dbpath           string
-	cvedbpath        string
-	cveDictionaryURL string
-
-	cvssScoreOver      float64
-	ignoreUnscoredCves bool
-
-	httpProxy       string
-	askSudoPassword bool
-	askKeyPassword  bool
-
-	// reporting
-	reportSlack     bool
-	reportMail      bool
-	reportJSON      bool
-	reportText      bool
-	reportS3        bool
-	reportAzureBlob bool
-
-	awsProfile  string
-	awsS3Bucket string
-	awsRegion   string
-
-	azureAccount   string
-	azureKey       string
-	azureContainer string
-
-	useYumPluginSecurity  bool
-	useUnattendedUpgrades bool
-
-	sshExternal bool
+	debug          bool
+	configPath     string
+	resultsDir     string
+	cacheDBPath    string
+	httpProxy      string
+	askKeyPassword bool
+	containersOnly bool
+	skipBroken     bool
+	sshExternal    bool
 }
 
 // Name return subcommand name
@@ -87,30 +57,15 @@ func (*ScanCmd) Synopsis() string { return "Scan vulnerabilities" }
 func (*ScanCmd) Usage() string {
 	return `scan:
 	scan
-		[-lang=en|ja]
 		[-config=/path/to/config.toml]
-		[-dbpath=/path/to/vuls.sqlite3]
-		[-cve-dictionary-dbpath=/path/to/cve.sqlite3]
-		[-cve-dictionary-url=http://127.0.0.1:1323]
-		[-cvss-over=7]
-		[-ignore-unscored-cves]
+		[-results-dir=/path/to/results]
+		[-cachedb-path=/path/to/cache.db]
 		[-ssh-external]
-		[-report-azure-blob]
-		[-report-json]
-		[-report-mail]
-		[-report-s3]
-		[-report-slack]
-		[-report-text]
+		[-containers-only]
+		[-skip-broken]
 		[-http-proxy=http://192.168.0.1:8080]
 		[-ask-key-password]
 		[-debug]
-		[-debug-sql]
-		[-aws-profile=default]
-		[-aws-region=us-west-2]
-		[-aws-s3-bucket=bucket_name]
-		[-azure-account=accout]
-		[-azure-key=key]
-		[-azure-container=container]
 
 		[SERVER]...
 `
@@ -118,42 +73,22 @@ func (*ScanCmd) Usage() string {
 
 // SetFlags set flag
 func (p *ScanCmd) SetFlags(f *flag.FlagSet) {
-	f.StringVar(&p.lang, "lang", "en", "[en|ja]")
 	f.BoolVar(&p.debug, "debug", false, "debug mode")
-	f.BoolVar(&p.debugSQL, "debug-sql", false, "SQL debug mode")
 
 	wd, _ := os.Getwd()
 
 	defaultConfPath := filepath.Join(wd, "config.toml")
 	f.StringVar(&p.configPath, "config", defaultConfPath, "/path/to/toml")
 
-	defaultDBPath := filepath.Join(wd, "vuls.sqlite3")
-	f.StringVar(&p.dbpath, "dbpath", defaultDBPath, "/path/to/sqlite3")
+	defaultResultsDir := filepath.Join(wd, "results")
+	f.StringVar(&p.resultsDir, "results-dir", defaultResultsDir, "/path/to/results")
 
+	defaultCacheDBPath := filepath.Join(wd, "cache.db")
 	f.StringVar(
-		&p.cvedbpath,
-		"cve-dictionary-dbpath",
-		"",
-		"/path/to/sqlite3 (For get cve detail from cve.sqlite3)")
-
-	defaultURL := "http://127.0.0.1:1323"
-	f.StringVar(
-		&p.cveDictionaryURL,
-		"cve-dictionary-url",
-		defaultURL,
-		"http://CVE.Dictionary")
-
-	f.Float64Var(
-		&p.cvssScoreOver,
-		"cvss-over",
-		0,
-		"-cvss-over=6.5 means reporting CVSS Score 6.5 and over (default: 0 (means report all))")
-
-	f.BoolVar(
-		&p.ignoreUnscoredCves,
-		"ignore-unscored-cves",
-		false,
-		"Don't report the unscored CVEs")
+		&p.cacheDBPath,
+		"cachedb-path",
+		defaultCacheDBPath,
+		"/path/to/cache.db (local cache of changelog for Ubuntu/Debian)")
 
 	f.BoolVar(
 		&p.sshExternal,
@@ -161,6 +96,18 @@ func (p *ScanCmd) SetFlags(f *flag.FlagSet) {
 		false,
 		"Use external ssh command. Default: Use the Go native implementation")
 
+	f.BoolVar(
+		&p.containersOnly,
+		"containers-only",
+		false,
+		"Scan containers only. Default: Scan both of hosts and containers")
+
+	f.BoolVar(
+		&p.skipBroken,
+		"skip-broken",
+		false,
+		"[For CentOS] yum update changelog with --skip-broken option")
+
 	f.StringVar(
 		&p.httpProxy,
 		"http-proxy",
@@ -168,65 +115,12 @@ func (p *ScanCmd) SetFlags(f *flag.FlagSet) {
 		"http://proxy-url:port (default: empty)",
 	)
 
-	f.BoolVar(&p.reportSlack, "report-slack", false, "Send report via Slack")
-	f.BoolVar(&p.reportMail, "report-mail", false, "Send report via Email")
-	f.BoolVar(&p.reportJSON,
-		"report-json",
-		false,
-		fmt.Sprintf("Write report to JSON files (%s/results/current)", wd),
-	)
-	f.BoolVar(&p.reportText,
-		"report-text",
-		false,
-		fmt.Sprintf("Write report to text files (%s/results/current)", wd),
-	)
-
-	f.BoolVar(&p.reportS3,
-		"report-s3",
-		false,
-		"Write report to S3 (bucket/yyyyMMdd_HHmm/servername.json)",
-	)
-	f.StringVar(&p.awsProfile, "aws-profile", "default", "AWS profile to use")
-	f.StringVar(&p.awsRegion, "aws-region", "us-east-1", "AWS region to use")
-	f.StringVar(&p.awsS3Bucket, "aws-s3-bucket", "", "S3 bucket name")
-
-	f.BoolVar(&p.reportAzureBlob,
-		"report-azure-blob",
-		false,
-		"Write report to S3 (container/yyyyMMdd_HHmm/servername.json)",
-	)
-	f.StringVar(&p.azureAccount, "azure-account", "", "Azure account name to use. AZURE_STORAGE_ACCOUNT environment variable is used if not specified")
-	f.StringVar(&p.azureKey, "azure-key", "", "Azure account key to use. AZURE_STORAGE_ACCESS_KEY environment variable is used if not specified")
-	f.StringVar(&p.azureContainer, "azure-container", "", "Azure storage container name")
-
 	f.BoolVar(
 		&p.askKeyPassword,
 		"ask-key-password",
 		false,
 		"Ask ssh privatekey password before scanning",
 	)
-
-	f.BoolVar(
-		&p.askSudoPassword,
-		"ask-sudo-password",
-		false,
-		"[Deprecated] THIS OPTION WAS REMOVED FOR SECURITY REASONS. Define NOPASSWD in /etc/sudoers on tareget servers and use SSH key-based authentication",
-	)
-
-	f.BoolVar(
-		&p.useYumPluginSecurity,
-		"use-yum-plugin-security",
-		false,
-		"[Deprecated] For CentOS 5. Scan by yum-plugin-security or not (use yum check-update by default)",
-	)
-
-	f.BoolVar(
-		&p.useUnattendedUpgrades,
-		"use-unattended-upgrades",
-		false,
-		"[Deprecated] For Ubuntu. Scan by unattended-upgrades or not (use apt-get upgrade --dry-run by default)",
-	)
-
 }
 
 // Execute execute
@@ -240,11 +134,8 @@ func (p *ScanCmd) Execute(_ context.Context, f *flag.FlagSet, _ ...interface{})
 			return subcommands.ExitFailure
 		}
 	}
-	if p.askSudoPassword {
-		logrus.Errorf("[Deprecated] -ask-sudo-password WAS REMOVED FOR SECURITY REASONS. Define NOPASSWD in /etc/sudoers on tareget servers and use SSH key-based authentication")
-		return subcommands.ExitFailure
-	}
 
+	c.Conf.Debug = p.debug
 	err = c.Load(p.configPath, keyPass)
 	if err != nil {
 		logrus.Errorf("Error loading %s, %s", p.configPath, err)
@@ -253,11 +144,6 @@ func (p *ScanCmd) Execute(_ context.Context, f *flag.FlagSet, _ ...interface{})
 
 	logrus.Info("Start scanning")
 	logrus.Infof("config: %s", p.configPath)
-	if p.cvedbpath != "" {
-		logrus.Infof("cve-dictionary: %s", p.cvedbpath)
-	} else {
-		logrus.Infof("cve-dictionary: %s", p.cveDictionaryURL)
-	}
 
 	var servernames []string
 	if 0 < len(f.Args()) {
@@ -295,90 +181,29 @@ func (p *ScanCmd) Execute(_ context.Context, f *flag.FlagSet, _ ...interface{})
 	if 0 < len(servernames) {
 		c.Conf.Servers = target
 	}
-
-	c.Conf.Lang = p.lang
-	c.Conf.Debug = p.debug
-	c.Conf.DebugSQL = p.debugSQL
+	logrus.Debugf("%s", pp.Sprintf("%v", target))
 
 	// logger
 	Log := util.NewCustomLogger(c.ServerInfo{})
 
-	// report
-	reports := []report.ResultWriter{
-		report.StdoutWriter{},
-		report.LogrusWriter{},
-	}
-	if p.reportSlack {
-		reports = append(reports, report.SlackWriter{})
-	}
-	if p.reportMail {
-		reports = append(reports, report.MailWriter{})
-	}
-	if p.reportJSON {
-		reports = append(reports, report.JSONWriter{})
-	}
-	if p.reportText {
-		reports = append(reports, report.TextFileWriter{})
-	}
-	if p.reportS3 {
-		c.Conf.AwsRegion = p.awsRegion
-		c.Conf.AwsProfile = p.awsProfile
-		c.Conf.S3Bucket = p.awsS3Bucket
-		if err := report.CheckIfBucketExists(); err != nil {
-			Log.Errorf("Failed to access to the S3 bucket. err: %s", err)
-			Log.Error("Ensure the bucket or check AWS config before scanning")
-			return subcommands.ExitUsageError
-		}
-		reports = append(reports, report.S3Writer{})
-	}
-	if p.reportAzureBlob {
-		c.Conf.AzureAccount = p.azureAccount
-		if c.Conf.AzureAccount == "" {
-			c.Conf.AzureAccount = os.Getenv("AZURE_STORAGE_ACCOUNT")
-		}
-
-		c.Conf.AzureKey = p.azureKey
-		if c.Conf.AzureKey == "" {
-			c.Conf.AzureKey = os.Getenv("AZURE_STORAGE_ACCESS_KEY")
-		}
-
-		c.Conf.AzureContainer = p.azureContainer
-		if c.Conf.AzureContainer == "" {
-			Log.Error("Azure storage container name is requied with --azure-container option")
-			return subcommands.ExitUsageError
-		}
-		if err := report.CheckIfAzureContainerExists(); err != nil {
-			Log.Errorf("Failed to access to the Azure Blob container. err: %s", err)
-			Log.Error("Ensure the container or check Azure config before scanning")
-			return subcommands.ExitUsageError
-		}
-		reports = append(reports, report.AzureBlobWriter{})
-	}
-
-	c.Conf.DBPath = p.dbpath
-	c.Conf.CveDBPath = p.cvedbpath
-	c.Conf.CveDictionaryURL = p.cveDictionaryURL
-	c.Conf.CvssScoreOver = p.cvssScoreOver
-	c.Conf.IgnoreUnscoredCves = p.ignoreUnscoredCves
+	c.Conf.ResultsDir = p.resultsDir
+	c.Conf.CacheDBPath = p.cacheDBPath
 	c.Conf.SSHExternal = p.sshExternal
 	c.Conf.HTTPProxy = p.httpProxy
-	c.Conf.UseYumPluginSecurity = p.useYumPluginSecurity
-	c.Conf.UseUnattendedUpgrades = p.useUnattendedUpgrades
+	c.Conf.ContainersOnly = p.containersOnly
+	c.Conf.SkipBroken = p.skipBroken
 
 	Log.Info("Validating Config...")
-	if !c.Conf.Validate() {
+	if !c.Conf.ValidateOnScan() {
 		return subcommands.ExitUsageError
 	}
 
-	if ok, err := cveapi.CveClient.CheckHealth(); !ok {
-		Log.Errorf("CVE HTTP server is not running. err: %s", err)
-		Log.Errorf("Run go-cve-dictionary as server mode or specify -cve-dictionary-dbpath option")
+	Log.Info("Detecting Server/Contianer OS... ")
+	if err := scan.InitServers(Log); err != nil {
+		Log.Errorf("Failed to init servers: %s", err)
 		return subcommands.ExitFailure
 	}
 
-	Log.Info("Detecting Server/Contianer OS... ")
-	scan.InitServers(Log)
-
 	Log.Info("Checking sudo configuration... ")
 	if err := scan.CheckIfSudoNoPasswd(Log); err != nil {
 		Log.Errorf("Failed to sudo with nopassword via SSH. Define NOPASSWD in /etc/sudoers on target servers")
@@ -395,36 +220,9 @@ func (p *ScanCmd) Execute(_ context.Context, f *flag.FlagSet, _ ...interface{})
 		}
 		return subcommands.ExitFailure
 	}
-
-	scanResults, err := scan.GetScanResults()
-	if err != nil {
-		Log.Fatal(err)
-		return subcommands.ExitFailure
-	}
-
-	Log.Info("Insert to DB...")
-	if err := db.OpenDB(); err != nil {
-		Log.Errorf("Failed to open DB. datafile: %s, err: %s", c.Conf.DBPath, err)
-		return subcommands.ExitFailure
-	}
-	if err := db.MigrateDB(); err != nil {
-		Log.Errorf("Failed to migrate. err: %s", err)
-		return subcommands.ExitFailure
-	}
-
-	if err := db.Insert(scanResults); err != nil {
-		Log.Fatalf("Failed to insert. dbpath: %s, err: %s", c.Conf.DBPath, err)
-		return subcommands.ExitFailure
-	}
-
-	Log.Info("Reporting...")
-	filtered := scanResults.FilterByCvssOver()
-	for _, w := range reports {
-		if err := w.Write(filtered); err != nil {
-			Log.Fatalf("Failed to report, err: %s", err)
-			return subcommands.ExitFailure
-		}
-	}
+	fmt.Printf("\n\n\n")
+	fmt.Println("To view the detail, vuls tui is useful.")
+	fmt.Println("To send a report, run vuls report -h.")
 
 	return subcommands.ExitSuccess
 }

commands/tui.go

@@ -18,26 +18,28 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
 package commands
 
 import (
+	"context"
 	"flag"
-	"fmt"
-	"io/ioutil"
 	"os"
 	"path/filepath"
-	"strconv"
-	"strings"
 
 	log "github.com/Sirupsen/logrus"
 	c "github.com/future-architect/vuls/config"
+	"github.com/future-architect/vuls/models"
 	"github.com/future-architect/vuls/report"
 	"github.com/google/subcommands"
-	"golang.org/x/net/context"
 )
 
 // TuiCmd is Subcommand of host discovery mode
 type TuiCmd struct {
-	lang     string
-	debugSQL bool
-	dbpath   string
+	lang       string
+	debugSQL   bool
+	resultsDir string
+
+	refreshCve       bool
+	cvedbtype        string
+	cvedbpath        string
+	cveDictionaryURL string
 }
 
 // Name return subcommand name
@@ -49,7 +51,13 @@ func (*TuiCmd) Synopsis() string { return "Run Tui view to anayze vulnerabilites
 // Usage return usage
 func (*TuiCmd) Usage() string {
 	return `tui:
-	tui [-dbpath=/path/to/vuls.sqlite3]
+	tui
+		[-cvedb-type=sqlite3|mysql]
+		[-cvedb-path=/path/to/cve.sqlite3]
+		[-cvedb-url=http://127.0.0.1:1323 or mysql connection string]
+		[-results-dir=/path/to/results]
+		[-refresh-cve]
+		[-debug-sql]
 
 `
 }
@@ -60,38 +68,87 @@ func (p *TuiCmd) SetFlags(f *flag.FlagSet) {
 	f.BoolVar(&p.debugSQL, "debug-sql", false, "debug SQL")
 
 	wd, _ := os.Getwd()
+	defaultResultsDir := filepath.Join(wd, "results")
+	f.StringVar(&p.resultsDir, "results-dir", defaultResultsDir, "/path/to/results")
 
-	defaultDBPath := filepath.Join(wd, "vuls.sqlite3")
-	f.StringVar(&p.dbpath, "dbpath", defaultDBPath,
-		fmt.Sprintf("/path/to/sqlite3 (default: %s)", defaultDBPath))
+	f.BoolVar(
+		&p.refreshCve,
+		"refresh-cve",
+		false,
+		"Refresh CVE information in JSON file under results dir")
+
+	f.StringVar(
+		&p.cvedbtype,
+		"cvedb-type",
+		"sqlite3",
+		"DB type for fetching CVE dictionary (sqlite3 or mysql)")
+
+	defaultCveDBPath := filepath.Join(wd, "cve.sqlite3")
+	f.StringVar(
+		&p.cvedbpath,
+		"cvedb-path",
+		defaultCveDBPath,
+		"/path/to/sqlite3 (For get cve detail from cve.sqlite3)")
+
+	f.StringVar(
+		&p.cveDictionaryURL,
+		"cvedb-url",
+		"",
+		"http://cve-dictionary.com:8080 or mysql connection string")
 }
 
 // Execute execute
 func (p *TuiCmd) Execute(_ context.Context, f *flag.FlagSet, _ ...interface{}) subcommands.ExitStatus {
 	c.Conf.Lang = "en"
 	c.Conf.DebugSQL = p.debugSQL
-	c.Conf.DBPath = p.dbpath
+	c.Conf.ResultsDir = p.resultsDir
+	c.Conf.CveDBType = p.cvedbtype
+	c.Conf.CveDBPath = p.cvedbpath
+	c.Conf.CveDictionaryURL = p.cveDictionaryURL
 
-	historyID := ""
-	if 0 < len(f.Args()) {
-		if _, err := strconv.Atoi(f.Args()[0]); err != nil {
-			log.Errorf("First Argument have to be scan_histores record ID: %s", err)
-			return subcommands.ExitFailure
-		}
-		historyID = f.Args()[0]
-	} else {
-		stat, _ := os.Stdin.Stat()
-		if (stat.Mode() & os.ModeCharDevice) == 0 {
-			bytes, err := ioutil.ReadAll(os.Stdin)
+	log.Info("Validating Config...")
+	if !c.Conf.ValidateOnTui() {
+		return subcommands.ExitUsageError
+	}
+
+	jsonDir, err := jsonDir(f.Args())
+	if err != nil {
+		log.Errorf("Failed to read json dir under results: %s", err)
+		return subcommands.ExitFailure
+	}
+
+	history, err := loadOneScanHistory(jsonDir)
+	if err != nil {
+		log.Errorf("Failed to read from JSON: %s", err)
+		return subcommands.ExitFailure
+	}
+
+	var results []models.ScanResult
+	for _, r := range history.ScanResults {
+		if p.refreshCve || needToRefreshCve(r) {
+			if c.Conf.CveDBType == "sqlite3" {
+				if _, err := os.Stat(c.Conf.CveDBPath); os.IsNotExist(err) {
+					log.Errorf("SQLite3 DB(CVE-Dictionary) is not exist: %s",
+						c.Conf.CveDBPath)
+					return subcommands.ExitFailure
+				}
+			}
+
+			filled, err := fillCveInfoFromCveDB(r)
 			if err != nil {
-				log.Errorf("Failed to read stdin: %s", err)
+				log.Errorf("Failed to fill CVE information: %s", err)
 				return subcommands.ExitFailure
 			}
-			fields := strings.Fields(string(bytes))
-			if 0 < len(fields) {
-				historyID = fields[0]
+
+			if err := overwriteJSONFile(jsonDir, filled); err != nil {
+				log.Errorf("Failed to write JSON: %s", err)
+				return subcommands.ExitFailure
 			}
+			results = append(results, filled)
+		} else {
+			results = append(results, r)
 		}
 	}
-	return report.RunTui(historyID)
+	history.ScanResults = results
+	return report.RunTui(history)
 }

commands/util.go

@@ -0,0 +1,225 @@
+/* Vuls - Vulnerability Scanner
+Copyright (C) 2016  Future Architect, Inc. Japan.
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+package commands
+
+import (
+	"encoding/json"
+	"fmt"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+	"regexp"
+	"sort"
+	"strings"
+
+	c "github.com/future-architect/vuls/config"
+	"github.com/future-architect/vuls/cveapi"
+	"github.com/future-architect/vuls/models"
+	"github.com/future-architect/vuls/report"
+	"github.com/future-architect/vuls/util"
+)
+
+// jsonDirPattern is file name pattern of JSON directory
+// 2016-11-16T10:43:28+09:00
+// 2016-11-16T10:43:28Z
+var jsonDirPattern = regexp.MustCompile(
+	`^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:Z|[+-]\d{2}:\d{2})$`)
+
+// JSONDirs is array of json files path.
+type jsonDirs []string
+
+// sort as recent directories are at the head
+func (d jsonDirs) Len() int {
+	return len(d)
+}
+func (d jsonDirs) Swap(i, j int) {
+	d[i], d[j] = d[j], d[i]
+}
+func (d jsonDirs) Less(i, j int) bool {
+	return d[j] < d[i]
+}
+
+// getValidJSONDirs return valid json directory as array
+// Returned array is sorted so that recent directories are at the head
+func lsValidJSONDirs() (dirs jsonDirs, err error) {
+	var dirInfo []os.FileInfo
+	if dirInfo, err = ioutil.ReadDir(c.Conf.ResultsDir); err != nil {
+		err = fmt.Errorf("Failed to read %s: %s", c.Conf.ResultsDir, err)
+		return
+	}
+	for _, d := range dirInfo {
+		if d.IsDir() && jsonDirPattern.MatchString(d.Name()) {
+			jsonDir := filepath.Join(c.Conf.ResultsDir, d.Name())
+			dirs = append(dirs, jsonDir)
+		}
+	}
+	sort.Sort(dirs)
+	return
+}
+
+// jsonDir returns
+// If there is an arg, check if it is a valid format and return the corresponding path under results.
+// If passed via PIPE (such as history subcommand), return that path.
+// Otherwise, returns the path of the latest directory
+func jsonDir(args []string) (string, error) {
+	var err error
+	var dirs jsonDirs
+
+	if 0 < len(args) {
+		if dirs, err = lsValidJSONDirs(); err != nil {
+			return "", err
+		}
+
+		path := filepath.Join(c.Conf.ResultsDir, args[0])
+		for _, d := range dirs {
+			ss := strings.Split(d, string(os.PathSeparator))
+			timedir := ss[len(ss)-1]
+			if timedir == args[0] {
+				return path, nil
+			}
+		}
+
+		return "", fmt.Errorf("Invalid path: %s", path)
+	}
+
+	// PIPE
+	stat, _ := os.Stdin.Stat()
+	if (stat.Mode() & os.ModeCharDevice) == 0 {
+		bytes, err := ioutil.ReadAll(os.Stdin)
+		if err != nil {
+			return "", fmt.Errorf("Failed to read stdin: %s", err)
+		}
+		fields := strings.Fields(string(bytes))
+		if 0 < len(fields) {
+			return filepath.Join(c.Conf.ResultsDir, fields[0]), nil
+		}
+		return "", fmt.Errorf("Stdin is invalid: %s", string(bytes))
+	}
+
+	// returns latest dir when no args or no PIPE
+	if dirs, err = lsValidJSONDirs(); err != nil {
+		return "", err
+	}
+	if len(dirs) == 0 {
+		return "", fmt.Errorf("No results under %s",
+			c.Conf.ResultsDir)
+	}
+	return dirs[0], nil
+}
+
+// loadOneScanHistory read JSON data
+func loadOneScanHistory(jsonDir string) (scanHistory models.ScanHistory, err error) {
+	var results []models.ScanResult
+	var files []os.FileInfo
+	if files, err = ioutil.ReadDir(jsonDir); err != nil {
+		err = fmt.Errorf("Failed to read %s: %s", jsonDir, err)
+		return
+	}
+	for _, f := range files {
+		if filepath.Ext(f.Name()) != ".json" {
+			continue
+		}
+		var r models.ScanResult
+		var data []byte
+		path := filepath.Join(jsonDir, f.Name())
+		if data, err = ioutil.ReadFile(path); err != nil {
+			err = fmt.Errorf("Failed to read %s: %s", path, err)
+			return
+		}
+		if json.Unmarshal(data, &r) != nil {
+			err = fmt.Errorf("Failed to parse %s: %s", path, err)
+			return
+		}
+		results = append(results, r)
+	}
+	if len(results) == 0 {
+		err = fmt.Errorf("There is no json file under %s", jsonDir)
+		return
+	}
+
+	scanHistory = models.ScanHistory{
+		ScanResults: results,
+	}
+	return
+}
+
+func fillCveInfoFromCveDB(r models.ScanResult) (filled models.ScanResult, err error) {
+	sInfo := c.Conf.Servers[r.ServerName]
+	vs, err := scanVulnByCpeNames(sInfo.CpeNames, r.ScannedCves)
+	if err != nil {
+		return
+	}
+	r.ScannedCves = vs
+	filled, err = r.FillCveDetail()
+	if err != nil {
+		return
+	}
+	return
+}
+
+func overwriteJSONFile(dir string, r models.ScanResult) error {
+	before := c.Conf.FormatJSON
+	c.Conf.FormatJSON = true
+	w := report.LocalFileWriter{CurrentDir: dir}
+	if err := w.Write(r); err != nil {
+		return fmt.Errorf("Failed to write summary report: %s", err)
+	}
+	c.Conf.FormatJSON = before
+	return nil
+}
+
+func scanVulnByCpeNames(cpeNames []string, scannedVulns []models.VulnInfo) ([]models.VulnInfo,
+	error) {
+	// To remove duplicate
+	set := map[string]models.VulnInfo{}
+	for _, v := range scannedVulns {
+		set[v.CveID] = v
+	}
+
+	for _, name := range cpeNames {
+		details, err := cveapi.CveClient.FetchCveDetailsByCpeName(name)
+		if err != nil {
+			return nil, err
+		}
+		for _, detail := range details {
+			if val, ok := set[detail.CveID]; ok {
+				names := val.CpeNames
+				names = util.AppendIfMissing(names, name)
+				val.CpeNames = names
+				set[detail.CveID] = val
+			} else {
+				set[detail.CveID] = models.VulnInfo{
+					CveID:    detail.CveID,
+					CpeNames: []string{name},
+				}
+			}
+		}
+	}
+
+	vinfos := []models.VulnInfo{}
+	for key := range set {
+		vinfos = append(vinfos, set[key])
+	}
+	return vinfos, nil
+}
+
+func needToRefreshCve(r models.ScanResult) bool {
+	return r.Lang != c.Conf.Lang || len(r.KnownCves) == 0 &&
+		len(r.UnknownCves) == 0 &&
+		len(r.IgnoredCves) == 0
+}

config/config.go

@@ -19,6 +19,7 @@ package config
 
 import (
 	"fmt"
+	"runtime"
 	"strings"
 
 	log "github.com/Sirupsen/logrus"
@@ -34,7 +35,7 @@ type Config struct {
 	DebugSQL bool
 	Lang     string
 
-	Mail    smtpConf
+	EMail   smtpConf
 	Slack   SlackConf
 	Default ServerInfo
 	Servers map[string]ServerInfo
@@ -44,11 +45,24 @@ type Config struct {
 	CvssScoreOver      float64
 	IgnoreUnscoredCves bool
 
-	SSHExternal bool
+	AssumeYes      bool
+	SSHExternal    bool
+	ContainersOnly bool
+	SkipBroken     bool
 
-	HTTPProxy string `valid:"url"`
-	DBPath    string
-	CveDBPath string
+	HTTPProxy   string `valid:"url"`
+	ResultsDir  string
+	CveDBType   string
+	CveDBPath   string
+	CacheDBPath string
+
+	FormatXML         bool
+	FormatJSON        bool
+	FormatOneLineText bool
+	FormatShortText   bool
+	FormatFullText    bool
+
+	GZIP bool
 
 	AwsProfile string
 	AwsRegion  string
@@ -57,28 +71,59 @@ type Config struct {
 	AzureAccount   string
 	AzureKey       string
 	AzureContainer string
-
-	//  CpeNames      []string
-	//  SummaryMode          bool
-	UseYumPluginSecurity  bool
-	UseUnattendedUpgrades bool
 }
 
-// Validate configuration
-func (c Config) Validate() bool {
+// ValidateOnConfigtest validates
+func (c Config) ValidateOnConfigtest() bool {
 	errs := []error{}
 
-	if len(c.DBPath) != 0 {
-		if ok, _ := valid.IsFilePath(c.DBPath); !ok {
+	if runtime.GOOS == "windows" && c.SSHExternal {
+		errs = append(errs, fmt.Errorf("-ssh-external cannot be used on windows"))
+	}
+
+	_, err := valid.ValidateStruct(c)
+	if err != nil {
+		errs = append(errs, err)
+	}
+
+	for _, err := range errs {
+		log.Error(err)
+	}
+
+	return len(errs) == 0
+}
+
+// ValidateOnPrepare validates configuration
+func (c Config) ValidateOnPrepare() bool {
+	return c.ValidateOnConfigtest()
+}
+
+// ValidateOnScan validates configuration
+func (c Config) ValidateOnScan() bool {
+	errs := []error{}
+
+	if len(c.ResultsDir) != 0 {
+		if ok, _ := valid.IsFilePath(c.ResultsDir); !ok {
 			errs = append(errs, fmt.Errorf(
-				"SQLite3 DB path must be a *Absolute* file path. dbpath: %s", c.DBPath))
+				"JSON base directory must be a *Absolute* file path. -results-dir: %s", c.ResultsDir))
 		}
 	}
 
-	if len(c.CveDBPath) != 0 {
-		if ok, _ := valid.IsFilePath(c.CveDBPath); !ok {
+	if runtime.GOOS == "windows" && c.SSHExternal {
+		errs = append(errs, fmt.Errorf("-ssh-external cannot be used on windows"))
+	}
+
+	if len(c.ResultsDir) != 0 {
+		if ok, _ := valid.IsFilePath(c.ResultsDir); !ok {
 			errs = append(errs, fmt.Errorf(
-				"SQLite3 DB(Cve Dictionary) path must be a *Absolute* file path. dbpath: %s", c.CveDBPath))
+				"JSON base directory must be a *Absolute* file path. -results-dir: %s", c.ResultsDir))
+		}
+	}
+
+	if len(c.CacheDBPath) != 0 {
+		if ok, _ := valid.IsFilePath(c.CacheDBPath); !ok {
+			errs = append(errs, fmt.Errorf(
+				"Cache DB path must be a *Absolute* file path. -cache-dbpath: %s", c.CacheDBPath))
 		}
 	}
 
@@ -87,7 +132,42 @@ func (c Config) Validate() bool {
 		errs = append(errs, err)
 	}
 
-	if mailerrs := c.Mail.Validate(); 0 < len(mailerrs) {
+	for _, err := range errs {
+		log.Error(err)
+	}
+
+	return len(errs) == 0
+}
+
+// ValidateOnReport validates configuration
+func (c Config) ValidateOnReport() bool {
+	errs := []error{}
+
+	if len(c.ResultsDir) != 0 {
+		if ok, _ := valid.IsFilePath(c.ResultsDir); !ok {
+			errs = append(errs, fmt.Errorf(
+				"JSON base directory must be a *Absolute* file path. -results-dir: %s", c.ResultsDir))
+		}
+	}
+
+	if c.CveDBType != "sqlite3" && c.CveDBType != "mysql" {
+		errs = append(errs, fmt.Errorf(
+			"CVE DB type must be either 'sqlite3' or 'mysql'.  -cve-dictionary-dbtype: %s", c.CveDBType))
+	}
+
+	if c.CveDBType == "sqlite3" {
+		if ok, _ := valid.IsFilePath(c.CveDBPath); !ok {
+			errs = append(errs, fmt.Errorf(
+				"SQLite3 DB(CVE-Dictionary) path must be a *Absolute* file path. -cve-dictionary-dbpath: %s", c.CveDBPath))
+		}
+	}
+
+	_, err := valid.ValidateStruct(c)
+	if err != nil {
+		errs = append(errs, err)
+	}
+
+	if mailerrs := c.EMail.Validate(); 0 < len(mailerrs) {
 		errs = append(errs, mailerrs...)
 	}
 
@@ -102,6 +182,36 @@ func (c Config) Validate() bool {
 	return len(errs) == 0
 }
 
+// ValidateOnTui validates configuration
+func (c Config) ValidateOnTui() bool {
+	errs := []error{}
+
+	if len(c.ResultsDir) != 0 {
+		if ok, _ := valid.IsFilePath(c.ResultsDir); !ok {
+			errs = append(errs, fmt.Errorf(
+				"JSON base directory must be a *Absolute* file path. -results-dir: %s", c.ResultsDir))
+		}
+	}
+
+	if c.CveDBType != "sqlite3" && c.CveDBType != "mysql" {
+		errs = append(errs, fmt.Errorf(
+			"CVE DB type must be either 'sqlite3' or 'mysql'.  -cve-dictionary-dbtype: %s", c.CveDBType))
+	}
+
+	if c.CveDBType == "sqlite3" {
+		if ok, _ := valid.IsFilePath(c.CveDBPath); !ok {
+			errs = append(errs, fmt.Errorf(
+				"SQLite3 DB(CVE-Dictionary) path must be a *Absolute* file path. -cve-dictionary-dbpath: %s", c.CveDBPath))
+		}
+	}
+
+	for _, err := range errs {
+		log.Error(err)
+	}
+
+	return len(errs) == 0
+}
+
 // smtpConf is smtp config
 type smtpConf struct {
 	SMTPAddr string
@@ -181,7 +291,6 @@ type SlackConf struct {
 
 // Validate validates configuration
 func (c *SlackConf) Validate() (errs []error) {
-
 	if !c.UseThisTime {
 		return
 	}
@@ -221,18 +330,43 @@ type ServerInfo struct {
 	KeyPath     string
 	KeyPassword string
 
-	CpeNames []string
+	CpeNames               []string
+	DependencyCheckXMLPath string
 
 	// Container Names or IDs
 	Containers []string
 
+	IgnoreCves []string
+
 	// Optional key-value set that will be outputted to JSON
 	Optional [][]interface{}
 
+	// For CentOS, RHEL, Amazon
+	Enablerepo string
+
 	// used internal
 	LogMsgAnsiColor string // DebugLog Color
 	Container       Container
-	Family          string
+	Distro          Distro
+}
+
+// GetServerName returns ServerName if this serverInfo is about host.
+// If this serverInfo is abount a container, returns containerID@ServerName
+func (s ServerInfo) GetServerName() string {
+	if len(s.Container.ContainerID) == 0 {
+		return s.ServerName
+	}
+	return fmt.Sprintf("%s@%s", s.Container.ContainerID, s.ServerName)
+}
+
+// Distro has distribution info
+type Distro struct {
+	Family  string
+	Release string
+}
+
+func (l Distro) String() string {
+	return fmt.Sprintf("%s %s", l.Family, l.Release)
 }
 
 // IsContainer returns whether this ServerInfo is about container

config/tomlloader.go

@@ -20,10 +20,11 @@ package config
 import (
 	"fmt"
 	"os"
+	"strings"
 
 	"github.com/BurntSushi/toml"
 	log "github.com/Sirupsen/logrus"
-	"github.com/k0kubun/pp"
+	"github.com/future-architect/vuls/contrib/owasp-dependency-check/parser"
 )
 
 // TOMLLoader loads config
@@ -31,14 +32,18 @@ type TOMLLoader struct {
 }
 
 // Load load the configuraiton TOML file specified by path arg.
-func (c TOMLLoader) Load(pathToToml, keyPass string) (err error) {
+func (c TOMLLoader) Load(pathToToml, keyPass string) error {
+	if Conf.Debug {
+		log.SetLevel(log.DebugLevel)
+	}
+
 	var conf Config
 	if _, err := toml.DecodeFile(pathToToml, &conf); err != nil {
 		log.Error("Load config failed", err)
 		return err
 	}
 
-	Conf.Mail = conf.Mail
+	Conf.EMail = conf.EMail
 	Conf.Slack = conf.Slack
 
 	d := conf.Default
@@ -51,7 +56,6 @@ func (c TOMLLoader) Load(pathToToml, keyPass string) (err error) {
 
 	i := 0
 	for name, v := range conf.Servers {
-
 		if 0 < len(v.KeyPassword) {
 			log.Warn("[Deprecated] KEYPASSWORD IN CONFIG FILE ARE UNSECURE. REMOVE THEM IMMEDIATELY FOR A SECURITY REASONS. THEY WILL BE REMOVED IN A FUTURE RELEASE.")
 		}
@@ -68,7 +72,7 @@ func (c TOMLLoader) Load(pathToToml, keyPass string) (err error) {
 		}
 
 		s.Host = v.Host
-		if s.Host == "" {
+		if len(s.Host) == 0 {
 			return fmt.Errorf("%s is invalid. host is empty", name)
 		}
 
@@ -82,7 +86,7 @@ func (c TOMLLoader) Load(pathToToml, keyPass string) (err error) {
 		}
 
 		s.KeyPath = v.KeyPath
-		if s.KeyPath == "" {
+		if len(s.KeyPath) == 0 {
 			s.KeyPath = d.KeyPath
 		}
 		if s.KeyPath != "" {
@@ -94,7 +98,7 @@ func (c TOMLLoader) Load(pathToToml, keyPass string) (err error) {
 
 		//  s.KeyPassword = keyPass
 		s.KeyPassword = v.KeyPassword
-		if s.KeyPassword == "" {
+		if len(s.KeyPassword) == 0 {
 			s.KeyPassword = d.KeyPassword
 		}
 
@@ -103,11 +107,42 @@ func (c TOMLLoader) Load(pathToToml, keyPass string) (err error) {
 			s.CpeNames = d.CpeNames
 		}
 
+		s.DependencyCheckXMLPath = v.DependencyCheckXMLPath
+		if len(s.DependencyCheckXMLPath) == 0 {
+			s.DependencyCheckXMLPath = d.DependencyCheckXMLPath
+		}
+
+		// Load CPEs from OWASP Dependency Check XML
+		if len(s.DependencyCheckXMLPath) != 0 {
+			cpes, err := parser.Parse(s.DependencyCheckXMLPath)
+			if err != nil {
+				return fmt.Errorf(
+					"Failed to read OWASP Dependency Check XML: %s", err)
+			}
+			log.Debugf("Loaded from OWASP Dependency Check XML: %s",
+				s.ServerName)
+			s.CpeNames = append(s.CpeNames, cpes...)
+		}
+
 		s.Containers = v.Containers
 		if len(s.Containers) == 0 {
 			s.Containers = d.Containers
 		}
 
+		s.IgnoreCves = v.IgnoreCves
+		for _, cve := range d.IgnoreCves {
+			found := false
+			for _, c := range s.IgnoreCves {
+				if cve == c {
+					found = true
+					break
+				}
+			}
+			if !found {
+				s.IgnoreCves = append(s.IgnoreCves, cve)
+			}
+		}
+
 		s.Optional = v.Optional
 		for _, dkv := range d.Optional {
 			found := false
@@ -122,13 +157,28 @@ func (c TOMLLoader) Load(pathToToml, keyPass string) (err error) {
 			}
 		}
 
+		s.Enablerepo = v.Enablerepo
+		if len(s.Enablerepo) == 0 {
+			s.Enablerepo = d.Enablerepo
+		}
+		if len(s.Enablerepo) != 0 {
+			for _, repo := range strings.Split(s.Enablerepo, ",") {
+				switch repo {
+				case "base", "updates":
+					// nop
+				default:
+					return fmt.Errorf(
+						"For now, enablerepo have to be base or updates: %s, servername: %s",
+						s.Enablerepo, name)
+				}
+			}
+		}
+
 		s.LogMsgAnsiColor = Colors[i%len(Colors)]
 		i++
 
 		servers[name] = s
 	}
-	log.Debug("Config loaded")
-	log.Debugf("%s", pp.Sprintf("%v", servers))
 	Conf.Servers = servers
-	return
+	return nil
 }

contrib/owasp-dependency-check/parser/parser.go

@@ -0,0 +1,64 @@
+package parser
+
+import (
+	"encoding/xml"
+	"fmt"
+	"io/ioutil"
+	"os"
+	"sort"
+	"strings"
+)
+
+type analysis struct {
+	Dependencies []dependency `xml:"dependencies>dependency"`
+}
+
+type dependency struct {
+	Identifiers []identifier `xml:"identifiers>identifier"`
+}
+
+type identifier struct {
+	Name string `xml:"name"`
+	Type string `xml:"type,attr"`
+}
+
+func appendIfMissing(slice []string, str string) []string {
+	for _, s := range slice {
+		if s == str {
+			return slice
+		}
+	}
+	return append(slice, str)
+}
+
+// Parse parses XML and collect list of cpe
+func Parse(path string) ([]string, error) {
+	file, err := os.Open(path)
+	if err != nil {
+		return []string{}, fmt.Errorf("Failed to open: %s", err)
+	}
+	defer file.Close()
+
+	b, err := ioutil.ReadAll(file)
+	if err != nil {
+		return []string{}, fmt.Errorf("Failed to read: %s", err)
+	}
+
+	var anal analysis
+	if err := xml.Unmarshal(b, &anal); err != nil {
+		fmt.Errorf("Failed to unmarshal: %s", err)
+	}
+
+	cpes := []string{}
+	for _, d := range anal.Dependencies {
+		for _, ident := range d.Identifiers {
+			if ident.Type == "cpe" {
+				name := strings.TrimPrefix(ident.Name, "(")
+				name = strings.TrimSuffix(name, ")")
+				cpes = appendIfMissing(cpes, name)
+			}
+		}
+	}
+	sort.Strings(cpes)
+	return cpes, nil
+}

cveapi/cve_client.go

@@ -48,8 +48,8 @@ func (api *cvedictClient) initialize() {
 }
 
 func (api cvedictClient) CheckHealth() (ok bool, err error) {
-	if config.Conf.CveDBPath != "" {
-		log.Debugf("get cve-dictionary from sqlite3")
+	if config.Conf.CveDictionaryURL == "" {
+		log.Debugf("get cve-dictionary from %s", config.Conf.CveDBType)
 		return true, nil
 	}
 
@@ -59,7 +59,7 @@ func (api cvedictClient) CheckHealth() (ok bool, err error) {
 	var resp *http.Response
 	resp, _, errs = gorequest.New().SetDebug(config.Conf.Debug).Get(url).End()
 	//  resp, _, errs = gorequest.New().Proxy(api.httpProxy).Get(url).End()
-	if len(errs) > 0 || resp == nil || resp.StatusCode != 200 {
+	if 0 < len(errs) || resp == nil || resp.StatusCode != 200 {
 		return false, fmt.Errorf("Failed to request to CVE server. url: %s, errs: %v", url, errs)
 	}
 	return true, nil
@@ -71,7 +71,7 @@ type response struct {
 }
 
 func (api cvedictClient) FetchCveDetails(cveIDs []string) (cveDetails cve.CveDetails, err error) {
-	if config.Conf.CveDBPath != "" {
+	if config.Conf.CveDictionaryURL == "" {
 		return api.FetchCveDetailsFromCveDB(cveIDs)
 	}
 
@@ -129,14 +129,19 @@ func (api cvedictClient) FetchCveDetails(cveIDs []string) (cveDetails cve.CveDet
 			fmt.Errorf("Failed to fetch CVE. err: %v", errs)
 	}
 
-	// order by CVE ID desc
 	sort.Sort(cveDetails)
 	return
 }
 
 func (api cvedictClient) FetchCveDetailsFromCveDB(cveIDs []string) (cveDetails cve.CveDetails, err error) {
-	log.Debugf("open cve-dictionary db")
-	cveconfig.Conf.DBPath = config.Conf.CveDBPath
+	log.Debugf("open cve-dictionary db (%s)", config.Conf.CveDBType)
+	cveconfig.Conf.DBType = config.Conf.CveDBType
+	if config.Conf.CveDBType == "sqlite3" {
+		cveconfig.Conf.DBPath = config.Conf.CveDBPath
+	} else {
+		cveconfig.Conf.DBPath = config.Conf.CveDictionaryURL
+	}
+	cveconfig.Conf.DebugSQL = config.Conf.DebugSQL
 	if err := cvedb.OpenDB(); err != nil {
 		return []cve.CveDetail{},
 			fmt.Errorf("Failed to open DB. err: %s", err)
@@ -186,48 +191,13 @@ func (api cvedictClient) httpGet(key, url string, resChan chan<- response, errCh
 	}
 }
 
-//  func (api cvedictClient) httpGet(key, url string, query map[string]string, resChan chan<- response, errChan chan<- error) {
-
-//      var body string
-//      var errs []error
-//      var resp *http.Response
-//      f := func() (err error) {
-//          req := gorequest.New().SetDebug(true).Proxy(api.httpProxy).Get(url)
-//          for key := range query {
-//              req = req.Query(fmt.Sprintf("%s=%s", key, query[key])).Set("Content-Type", "application/x-www-form-urlencoded")
-//          }
-//          pp.Println(req)
-//          resp, body, errs = req.End()
-//          if len(errs) > 0 || resp.StatusCode != 200 {
-//              errChan <- fmt.Errorf("HTTP error. errs: %v, url: %s", errs, url)
-//          }
-//          return nil
-//      }
-//      notify := func(err error, t time.Duration) {
-//          log.Warnf("Failed to get. retrying in %s seconds. err: %s", t, err)
-//      }
-//      err := backoff.RetryNotify(f, backoff.NewExponentialBackOff(), notify)
-//      if err != nil {
-//          errChan <- fmt.Errorf("HTTP Error %s", err)
-//      }
-//      //  resChan <- body
-//      cveDetail := cve.CveDetail{}
-//      if err := json.Unmarshal([]byte(body), &cveDetail); err != nil {
-//          errChan <- fmt.Errorf("Failed to Unmarshall. body: %s, err: %s", body, err)
-//      }
-//      resChan <- response{
-//          key,
-//          cveDetail,
-//      }
-//  }
-
 type responseGetCveDetailByCpeName struct {
 	CpeName    string
 	CveDetails []cve.CveDetail
 }
 
 func (api cvedictClient) FetchCveDetailsByCpeName(cpeName string) ([]cve.CveDetail, error) {
-	if config.Conf.CveDBPath != "" {
+	if config.Conf.CveDictionaryURL == "" {
 		return api.FetchCveDetailsByCpeNameFromDB(cpeName)
 	}
 
@@ -252,7 +222,7 @@ func (api cvedictClient) httpPost(key, url string, query map[string]string) ([]c
 			req = req.Send(fmt.Sprintf("%s=%s", key, query[key])).Type("json")
 		}
 		resp, body, errs = req.End()
-		if len(errs) > 0 || resp == nil || resp.StatusCode != 200 {
+		if 0 < len(errs) || resp == nil || resp.StatusCode != 200 {
 			return fmt.Errorf("HTTP POST error: %v, url: %s, resp: %v", errs, url, resp)
 		}
 		return nil
@@ -274,8 +244,11 @@ func (api cvedictClient) httpPost(key, url string, query map[string]string) ([]c
 }
 
 func (api cvedictClient) FetchCveDetailsByCpeNameFromDB(cpeName string) ([]cve.CveDetail, error) {
-	log.Debugf("open cve-dictionary db")
+	log.Debugf("open cve-dictionary db (%s)", config.Conf.CveDBType)
+	cveconfig.Conf.DBType = config.Conf.CveDBType
 	cveconfig.Conf.DBPath = config.Conf.CveDBPath
+	cveconfig.Conf.DebugSQL = config.Conf.DebugSQL
+
 	if err := cvedb.OpenDB(); err != nil {
 		return []cve.CveDetail{},
 			fmt.Errorf("Failed to open DB. err: %s", err)

db/db.go

@@ -1,324 +0,0 @@
-/* Vuls - Vulnerability Scanner
-Copyright (C) 2016  Future Architect, Inc. Japan.
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-package db
-
-import (
-	"fmt"
-	"sort"
-	"strconv"
-	"time"
-
-	"github.com/future-architect/vuls/config"
-	m "github.com/future-architect/vuls/models"
-	"github.com/jinzhu/gorm"
-	cvedb "github.com/kotakanbe/go-cve-dictionary/db"
-	cve "github.com/kotakanbe/go-cve-dictionary/models"
-)
-
-var db *gorm.DB
-
-// OpenDB opens Database
-func OpenDB() (err error) {
-	db, err = gorm.Open("sqlite3", config.Conf.DBPath)
-	if err != nil {
-		err = fmt.Errorf("Failed to open DB. datafile: %s, err: %s", config.Conf.DBPath, err)
-		return
-
-	}
-	db.LogMode(config.Conf.DebugSQL)
-	return
-}
-
-// MigrateDB migrates Database
-func MigrateDB() error {
-	if err := db.AutoMigrate(
-		&m.ScanHistory{},
-		&m.ScanResult{},
-		//  &m.NWLink{},
-		&m.Container{},
-		&m.CveInfo{},
-		&m.CpeName{},
-		&m.PackageInfo{},
-		&m.DistroAdvisory{},
-		&cve.CveDetail{},
-		&cve.Jvn{},
-		&cve.Nvd{},
-		&cve.Reference{},
-		&cve.Cpe{},
-	).Error; err != nil {
-		return fmt.Errorf("Failed to migrate. err: %s", err)
-	}
-
-	errMsg := "Failed to create index. err: %s"
-	//  if err := db.Model(&m.NWLink{}).
-	//      AddIndex("idx_n_w_links_scan_result_id", "scan_result_id").Error; err != nil {
-	//      return fmt.Errorf(errMsg, err)
-	//  }
-	if err := db.Model(&m.Container{}).
-		AddIndex("idx_containers_scan_result_id", "scan_result_id").Error; err != nil {
-		return fmt.Errorf(errMsg, err)
-	}
-	if err := db.Model(&m.CveInfo{}).
-		AddIndex("idx_cve_infos_scan_result_id", "scan_result_id").Error; err != nil {
-		return fmt.Errorf(errMsg, err)
-	}
-	if err := db.Model(&m.CpeName{}).
-		AddIndex("idx_cpe_names_cve_info_id", "cve_info_id").Error; err != nil {
-		return fmt.Errorf(errMsg, err)
-	}
-	if err := db.Model(&m.PackageInfo{}).
-		AddIndex("idx_package_infos_cve_info_id", "cve_info_id").Error; err != nil {
-		return fmt.Errorf(errMsg, err)
-	}
-	if err := db.Model(&m.DistroAdvisory{}).
-		//TODO check table name
-		AddIndex("idx_distro_advisories_cve_info_id", "cve_info_id").Error; err != nil {
-		return fmt.Errorf(errMsg, err)
-	}
-	if err := db.Model(&cve.CveDetail{}).
-		AddIndex("idx_cve_details_cve_info_id", "cve_info_id").Error; err != nil {
-		return fmt.Errorf(errMsg, err)
-	}
-	if err := db.Model(&cve.CveDetail{}).
-		AddIndex("idx_cve_details_cveid", "cve_id").Error; err != nil {
-		return fmt.Errorf(errMsg, err)
-	}
-	if err := db.Model(&cve.Nvd{}).
-		AddIndex("idx_nvds_cve_detail_id", "cve_detail_id").Error; err != nil {
-		return fmt.Errorf(errMsg, err)
-	}
-	if err := db.Model(&cve.Jvn{}).
-		AddIndex("idx_jvns_cve_detail_id", "cve_detail_id").Error; err != nil {
-		return fmt.Errorf(errMsg, err)
-	}
-	if err := db.Model(&cve.Cpe{}).
-		AddIndex("idx_cpes_jvn_id", "jvn_id").Error; err != nil {
-		return fmt.Errorf(errMsg, err)
-	}
-	if err := db.Model(&cve.Reference{}).
-		AddIndex("idx_references_jvn_id", "jvn_id").Error; err != nil {
-		return fmt.Errorf(errMsg, err)
-	}
-	if err := db.Model(&cve.Cpe{}).
-		AddIndex("idx_cpes_nvd_id", "nvd_id").Error; err != nil {
-		return fmt.Errorf(errMsg, err)
-	}
-	if err := db.Model(&cve.Reference{}).
-		AddIndex("idx_references_nvd_id", "nvd_id").Error; err != nil {
-		return fmt.Errorf(errMsg, err)
-	}
-
-	return nil
-}
-
-// Insert inserts scan results into DB
-func Insert(results []m.ScanResult) error {
-	for _, r := range results {
-		r.KnownCves = resetGormIDs(r.KnownCves)
-		r.UnknownCves = resetGormIDs(r.UnknownCves)
-	}
-
-	history := m.ScanHistory{
-		ScanResults: results,
-		ScannedAt:   time.Now(),
-	}
-
-	db = db.Set("gorm:save_associations", false)
-	if err := db.Create(&history).Error; err != nil {
-		return err
-	}
-	for _, scanResult := range history.ScanResults {
-		scanResult.ScanHistoryID = history.ID
-		if err := db.Create(&scanResult).Error; err != nil {
-			return err
-		}
-		scanResult.Container.ScanResultID = scanResult.ID
-		if err := db.Create(&scanResult.Container).Error; err != nil {
-			return err
-		}
-		if err := insertCveInfos(scanResult.ID, scanResult.KnownCves); err != nil {
-			return err
-		}
-		if err := insertCveInfos(scanResult.ID, scanResult.UnknownCves); err != nil {
-			return err
-		}
-	}
-	return nil
-}
-
-func insertCveInfos(scanResultID uint, infos []m.CveInfo) error {
-	for _, cveInfo := range infos {
-		cveInfo.ScanResultID = scanResultID
-		if err := db.Create(&cveInfo).Error; err != nil {
-			return err
-		}
-
-		for _, pack := range cveInfo.Packages {
-			pack.CveInfoID = cveInfo.ID
-			if err := db.Create(&pack).Error; err != nil {
-				return err
-			}
-		}
-
-		for _, distroAdvisory := range cveInfo.DistroAdvisories {
-			distroAdvisory.CveInfoID = cveInfo.ID
-			if err := db.Create(&distroAdvisory).Error; err != nil {
-				return err
-			}
-		}
-
-		for _, cpeName := range cveInfo.CpeNames {
-			cpeName.CveInfoID = cveInfo.ID
-			if err := db.Create(&cpeName).Error; err != nil {
-				return err
-			}
-		}
-
-		db = db.Set("gorm:save_associations", true)
-		cveDetail := cveInfo.CveDetail
-		cveDetail.CveInfoID = cveInfo.ID
-		if err := db.Create(&cveDetail).Error; err != nil {
-			return err
-		}
-		db = db.Set("gorm:save_associations", false)
-	}
-	return nil
-}
-
-func resetGormIDs(infos []m.CveInfo) []m.CveInfo {
-	for i := range infos {
-		infos[i].CveDetail.ID = 0
-		// NVD
-		infos[i].CveDetail.Nvd.ID = 0
-		for j := range infos[i].CveDetail.Nvd.Cpes {
-			infos[i].CveDetail.Nvd.Cpes[j].ID = 0
-		}
-		for j := range infos[i].CveDetail.Nvd.References {
-			infos[i].CveDetail.Nvd.References[j].ID = 0
-		}
-
-		// JVN
-		infos[i].CveDetail.Jvn.ID = 0
-		for j := range infos[i].CveDetail.Jvn.Cpes {
-			infos[i].CveDetail.Jvn.Cpes[j].ID = 0
-		}
-		for j := range infos[i].CveDetail.Jvn.References {
-			infos[i].CveDetail.Jvn.References[j].ID = 0
-		}
-
-		//Packages
-		for j := range infos[i].Packages {
-			infos[i].Packages[j].ID = 0
-			infos[i].Packages[j].CveInfoID = 0
-		}
-	}
-	return infos
-}
-
-// SelectScanHistory select scan history from DB
-func SelectScanHistory(historyID string) (m.ScanHistory, error) {
-	var err error
-
-	scanHistory := m.ScanHistory{}
-	if historyID == "" {
-		// select latest
-		db.Order("scanned_at desc").First(&scanHistory)
-	} else {
-		var id int
-		if id, err = strconv.Atoi(historyID); err != nil {
-			return m.ScanHistory{},
-				fmt.Errorf("historyID have to be numeric number: %s", err)
-		}
-		db.First(&scanHistory, id)
-	}
-
-	if scanHistory.ID == 0 {
-		return m.ScanHistory{}, fmt.Errorf("No scanHistory records")
-	}
-
-	//  results := []m.ScanResult{}
-	results := m.ScanResults{}
-	db.Model(&scanHistory).Related(&results, "ScanResults")
-	scanHistory.ScanResults = results
-
-	for i, r := range results {
-		//  nw := []m.NWLink{}
-		//  db.Model(&r).Related(&nw, "NWLinks")
-		//  scanHistory.ScanResults[i].NWLinks = nw
-
-		di := m.Container{}
-		db.Model(&r).Related(&di, "Container")
-		scanHistory.ScanResults[i].Container = di
-
-		knownCves := selectCveInfos(&r, "KnownCves")
-		sort.Sort(m.CveInfos(knownCves))
-		scanHistory.ScanResults[i].KnownCves = knownCves
-	}
-
-	sort.Sort(scanHistory.ScanResults)
-	return scanHistory, nil
-}
-
-func selectCveInfos(result *m.ScanResult, fieldName string) []m.CveInfo {
-	cveInfos := []m.CveInfo{}
-	db.Model(&result).Related(&cveInfos, fieldName)
-
-	for i, cveInfo := range cveInfos {
-		cveDetail := cve.CveDetail{}
-		db.Model(&cveInfo).Related(&cveDetail, "CveDetail")
-		id := cveDetail.CveID
-		filledCveDetail := cvedb.Get(id, db)
-		cveInfos[i].CveDetail = filledCveDetail
-
-		packs := []m.PackageInfo{}
-		db.Model(&cveInfo).Related(&packs, "Packages")
-		cveInfos[i].Packages = packs
-
-		advisories := []m.DistroAdvisory{}
-		db.Model(&cveInfo).Related(&advisories, "DistroAdvisories")
-		cveInfos[i].DistroAdvisories = advisories
-
-		names := []m.CpeName{}
-		db.Model(&cveInfo).Related(&names, "CpeNames")
-		cveInfos[i].CpeNames = names
-	}
-	return cveInfos
-}
-
-// SelectScanHistories select latest scan history from DB
-func SelectScanHistories() ([]m.ScanHistory, error) {
-	scanHistories := []m.ScanHistory{}
-	db.Order("scanned_at desc").Find(&scanHistories)
-
-	if len(scanHistories) == 0 {
-		return []m.ScanHistory{}, fmt.Errorf("No scanHistory records")
-	}
-
-	for i, history := range scanHistories {
-		results := m.ScanResults{}
-		db.Model(&history).Related(&results, "ScanResults")
-		scanHistories[i].ScanResults = results
-
-		for j, r := range results {
-			di := m.Container{}
-			db.Model(&r).Related(&di, "Container")
-			scanHistories[i].ScanResults[j].Container = di
-		}
-	}
-	return scanHistories, nil
-}

glide.lock

@@ -1,117 +1,123 @@
-hash: 9683c87b3cf998e7fac1b12c4a94bf2bd18cb5422e9108539811546e703a439a
-updated: 2016-07-12T16:20:45.462913061+09:00
+hash: c3167d83e68562cd7ef73f138ce60cb9e60b72b50394e8615388d1f3ba9fbef2
+updated: 2017-01-02T09:37:09.437363123+09:00
 imports:
 - name: github.com/asaskevich/govalidator
-  version: df81827fdd59d8b4fb93d8910b286ab7a3919520
+  version: 7b3beb6df3c42abd3509abfc3bcacc0fbfb7c877
 - name: github.com/aws/aws-sdk-go
-  version: 90dec2183a5f5458ee79cbaf4b8e9ab910bc81a6
+  version: 5b341290c488aa6bd76b335d819b4a68516ec3ab
   subpackages:
   - aws
-  - aws/credentials
-  - aws/session
-  - service/s3
   - aws/awserr
-  - aws/client
-  - aws/corehandlers
-  - aws/defaults
-  - aws/request
-  - private/endpoints
   - aws/awsutil
+  - aws/client
   - aws/client/metadata
-  - aws/signer/v4
-  - private/protocol
-  - private/protocol/restxml
-  - private/waiter
+  - aws/corehandlers
+  - aws/credentials
   - aws/credentials/ec2rolecreds
+  - aws/credentials/endpointcreds
+  - aws/credentials/stscreds
+  - aws/defaults
   - aws/ec2metadata
-  - private/protocol/rest
+  - aws/request
+  - aws/session
+  - aws/signer/v4
+  - private/endpoints
+  - private/protocol
   - private/protocol/query
-  - private/protocol/xml/xmlutil
   - private/protocol/query/queryutil
+  - private/protocol/rest
+  - private/protocol/restxml
+  - private/protocol/xml/xmlutil
+  - private/waiter
+  - service/s3
+  - service/sts
 - name: github.com/Azure/azure-sdk-for-go
-  version: 58a13e378daf3b06e65925397185684b16321111
+  version: 27ae5c8b5bc5d90ab0540b4c5d0f2632c8db8b57
   subpackages:
   - storage
+- name: github.com/boltdb/bolt
+  version: 315c65d4cf4f5278c73477a35fb1f9b08365d340
 - name: github.com/BurntSushi/toml
-  version: ffaa107fbd880f6d18cd6fec9b511668dcad8639
+  version: 99064174e013895bbd9b025c31100bd1d9b590ca
 - name: github.com/cenkalti/backoff
-  version: cdf48bbc1eb78d1349cbda326a4a037f7ba565c6
+  version: 8edc80b07f38c27352fb186d971c628a6c32552b
 - name: github.com/cheggaaa/pb
-  version: 04b234c80d661c663dbcebd52fc7218fdacc6d0c
+  version: ad4efe000aa550bb54918c06ebbadc0ff17687b9
 - name: github.com/go-ini/ini
-  version: cf53f9204df4fbdd7ec4164b57fa6184ba168292
+  version: 6e4869b434bd001f6983749881c7ead3545887d8
+- name: github.com/go-sql-driver/mysql
+  version: d512f204a577a4ab037a1816604c48c9c13210be
 - name: github.com/google/subcommands
-  version: 1c7173745a6001f67d8d96ab4e178284c77f7759
+  version: a71b91e238406bd68766ee52db63bebedce0e9f6
 - name: github.com/gosuri/uitable
   version: 36ee7e946282a3fb1cfecd476ddc9b35d8847e42
   subpackages:
   - util/strutil
   - util/wordwrap
 - name: github.com/howeyc/gopass
-  version: 66487b23f2880ba32e185121d2cd51a338ea069a
+  version: f5387c492211eb133053880d23dfae62aa14123d
 - name: github.com/jinzhu/gorm
-  version: 613c0655691abb7691b70c5fda80a716d9e20b1b
+  version: 39165d498058a823126af3cbf4d2a3b0e1acf11e
+  subpackages:
+  - dialects/mysql
+  - dialects/sqlite
 - name: github.com/jinzhu/inflection
-  version: 8f4d3a0d04ce0b7c0cf3126fb98524246d00d102
+  version: 74387dc39a75e970e7a3ae6a3386b5bd2e5c5cff
 - name: github.com/jmespath/go-jmespath
-  version: 0b12d6b521d83fc7f755e7cfc1b1fbdd35a01a74
+  version: bd40a432e4c76585ef6b72d3fd96fb9b6dc7b68d
 - name: github.com/jroimartin/gocui
-  version: 2dcda558bf18ec07c7065bf1eaf071b5305f7c0c
+  version: ba396278de0a3c63658bbaba13d2d2fa392edb11
 - name: github.com/k0kubun/pp
   version: f5dce6ed0ccf6c350f1679964ff6b61f3d6d2033
 - name: github.com/kotakanbe/go-cve-dictionary
-  version: 1a336b8ac785badfe89a175ee926d39574901232
+  version: 7eb1f1a2e7e436177570bf234e21c2ed9489d3fb
   subpackages:
   - config
   - db
-  - models
-  - log
   - jvn
+  - log
+  - models
   - nvd
+  - util
 - name: github.com/kotakanbe/go-pingscanner
   version: 58e188a3e4f6ab1a6371e33421e4502e26fa1e80
 - name: github.com/kotakanbe/logrus-prefixed-formatter
   version: f4f7d41649cf1e75e736884da8d05324aa76ea25
 - name: github.com/mattn/go-colorable
-  version: 9056b7a9f2d1f2d96498d6d146acd1f9d5ed3d59
+  version: 6c903ff4aa50920ca86087a280590b36b3152b9c
 - name: github.com/mattn/go-isatty
-  version: 56b76bdf51f7708750eac80fa38b952bb9f32639
+  version: 66b8e73f3f5cda9f96b69efd03dd3d7fc4a5cdb8
 - name: github.com/mattn/go-runewidth
-  version: d6bea18f789704b5f83375793155289da36a3c7f
+  version: 737072b4e32b7a5018b4a7125da8d12de90e8045
 - name: github.com/mattn/go-sqlite3
-  version: 38ee283dabf11c9cbdb968eebd79b1fa7acbabe6
+  version: 5510da399572b4962c020184bb291120c0a412e2
 - name: github.com/mgutz/ansi
   version: c286dcecd19ff979eeb73ea444e479b903f2cfcb
 - name: github.com/moul/http2curl
   version: b1479103caacaa39319f75e7f57fc545287fca0d
 - name: github.com/nsf/termbox-go
-  version: c45773466a30b680355d6494cc8826113c93cd0f
+  version: b6acae516ace002cb8105a89024544a1480655a5
 - name: github.com/parnurzeal/gorequest
-  version: 6e8ad4ebdee4bec2934ed5afaaa1c7b877832a17
+  version: e37b9d1efacf7c94820b29b75dd7d0c2996b3fb1
 - name: github.com/rifflock/lfshook
-  version: 05a24e24fa8d3a2eca8c2baf23aa2d5a2c51490c
+  version: 3f9d976bd7402de39b46357069fb6325a974572e
 - name: github.com/Sirupsen/logrus
-  version: f3cfb454f4c209e6668c95216c4744b8fddb2356
+  version: 3ec0642a7fb6488f65b06f9040adc67e3990296a
 - name: golang.org/x/crypto
-  version: c2f4947f41766b144bb09066e919466da5eddeae
+  version: 9477e0b78b9ac3d0b03822fd95422e2fe07627cd
   subpackages:
-  - ssh
-  - ssh/agent
-  - ssh/terminal
   - curve25519
   - ed25519
   - ed25519/internal/edwards25519
+  - ssh
+  - ssh/agent
+  - ssh/terminal
 - name: golang.org/x/net
-  version: f841c39de738b1d0df95b5a7187744f0e03d8112
+  version: 1d7a0b2100da090d8b02afcfb42f97e2c77e71a4
   subpackages:
-  - context
   - publicsuffix
 - name: golang.org/x/sys
-  version: a408501be4d17ee978c04a618e7a1b22af058c0e
+  version: 9bb9f0998d48b31547d975974935ae9b48c7a03c
   subpackages:
   - unix
-- name: gopkg.in/alexcesaro/quotedprintable.v3
-  version: 2caba252f4dc53eaf6b553000885530023f54623
-- name: gopkg.in/gomail.v2
-  version: 81ebce5c23dfd25c6c67194b37d3dd3f338c98b1
-devImports: []
+testImports: []

glide.yaml

@@ -12,17 +12,18 @@ import:
   - aws/credentials
   - aws/session
   - service/s3
+- package: github.com/boltdb/bolt
 - package: github.com/cenkalti/backoff
 - package: github.com/google/subcommands
 - package: github.com/gosuri/uitable
 - package: github.com/howeyc/gopass
-- package: github.com/jinzhu/gorm
 - package: github.com/jroimartin/gocui
 - package: github.com/k0kubun/pp
 - package: github.com/kotakanbe/go-cve-dictionary
   subpackages:
   - config
   - db
+  - log
   - models
 - package: github.com/kotakanbe/go-pingscanner
 - package: github.com/kotakanbe/logrus-prefixed-formatter
@@ -33,7 +34,3 @@ import:
   subpackages:
   - ssh
   - ssh/agent
-- package: golang.org/x/net
-  subpackages:
-  - context
-- package: gopkg.in/gomail.v2

img/vuls-scan-flow.graphml

@@ -72,7 +72,7 @@ FreeBSD: pkg<y:LabelModel>
           <y:Geometry height="56.0" width="268.0" x="10.0" y="287.8409153761062"/>
           <y:Fill color="#E8EEF7" color2="#B7C9E3" transparent="false"/>
           <y:BorderStyle color="#000000" type="line" width="1.0"/>
-          <y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Dialog" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="32.265625" modelName="custom" textColor="#000000" visible="true" width="260.83984375" x="3.580078125" y="11.8671875">Get upgradable packages
+          <y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Dialog" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="32.265625" modelName="custom" textColor="#000000" visible="true" width="260.83984375" x="3.580078125" y="11.8671875">Check upgradable packages
 Debian/Ubuntu: apt-get upgrade --dry-run<y:LabelModel>
               <y:SmartNodeLabelModel distance="4.0"/>
             </y:LabelModel>
@@ -168,22 +168,6 @@ FreeBSD: pkg audit<y:LabelModel>
       </data>
     </node>
     <node id="n9">
-      <data key="d6">
-        <y:GenericNode configuration="com.yworks.flowchart.dataBase">
-          <y:Geometry height="64.1719342604298" width="111.96965865992411" x="687.3850119398792" y="807.0697396491782"/>
-          <y:Fill color="#E8EEF7" color2="#B7C9E3" transparent="false"/>
-          <y:BorderStyle color="#000000" type="line" width="1.0"/>
-          <y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Dialog" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="18.1328125" modelName="custom" textColor="#000000" visible="true" width="48.56640625" x="31.701626204962054" y="23.019560880214726">Vuls DB<y:LabelModel>
-              <y:SmartNodeLabelModel distance="4.0"/>
-            </y:LabelModel>
-            <y:ModelParameter>
-              <y:SmartNodeLabelModelParameter labelRatioX="0.0" labelRatioY="0.0" nodeRatioX="0.0" nodeRatioY="-8.881784197001252E-16" offsetX="0.0" offsetY="0.0" upX="0.0" upY="-1.0"/>
-            </y:ModelParameter>
-          </y:NodeLabel>
-        </y:GenericNode>
-      </data>
-    </node>
-    <node id="n10">
       <data key="d6">
         <y:GenericNode configuration="com.yworks.flowchart.dataBase">
           <y:Geometry height="65.22882427307195" width="136.83944374209864" x="411.5802781289507" y="687.385587863464"/>
@@ -199,13 +183,13 @@ FreeBSD: pkg audit<y:LabelModel>
         </y:GenericNode>
       </data>
     </node>
-    <node id="n11">
+    <node id="n10">
       <data key="d6">
         <y:GenericNode configuration="com.yworks.flowchart.process">
           <y:Geometry height="56.0" width="268.0" x="609.3698412698412" y="716.4553275126422"/>
           <y:Fill color="#E8EEF7" color2="#B7C9E3" transparent="false"/>
           <y:BorderStyle color="#000000" type="line" width="1.0"/>
-          <y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Dialog" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="32.265625" modelName="custom" textColor="#000000" visible="true" width="126.396484375" x="70.8017578125" y="11.8671875">Insert results into DB
+          <y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Dialog" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="32.265625" modelName="custom" textColor="#000000" visible="true" width="152.634765625" x="57.6826171875" y="11.8671875">Write results to JSON files
 Reporting<y:LabelModel>
               <y:SmartNodeLabelModel distance="4.0"/>
             </y:LabelModel>
@@ -216,13 +200,13 @@ Reporting<y:LabelModel>
         </y:GenericNode>
       </data>
     </node>
-    <node id="n12">
+    <node id="n11">
       <data key="d6">
         <y:GenericNode configuration="com.yworks.flowchart.process">
           <y:Geometry height="56.0" width="268.0" x="309.6849206349206" y="287.8409153761062"/>
           <y:Fill color="#E8EEF7" color2="#B7C9E3" transparent="false"/>
           <y:BorderStyle color="#000000" type="line" width="1.0"/>
-          <y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Dialog" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="32.265625" modelName="custom" textColor="#000000" visible="true" width="271.369140625" x="-1.6845703124999432" y="11.8671875">Get all changelogs by using package manager
+          <y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Dialog" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="32.265625" modelName="custom" textColor="#000000" visible="true" width="293.06640625" x="-12.533203124999943" y="11.8671875">Get all changelogs of updatable packages at once
 CentOS: yum update --changelog<y:LabelModel>
               <y:SmartNodeLabelModel distance="4.0"/>
             </y:LabelModel>
@@ -233,7 +217,7 @@ CentOS: yum update --changelog<y:LabelModel>
         </y:GenericNode>
       </data>
     </node>
-    <node id="n13">
+    <node id="n12">
       <data key="d6">
         <y:GenericNode configuration="com.yworks.flowchart.process">
           <y:Geometry height="56.0" width="268.0" x="309.6849206349206" y="373.8409153761062"/>
@@ -364,7 +348,7 @@ FreeBSD<y:LabelModel>
         </y:PolyLineEdge>
       </data>
     </edge>
-    <edge id="e9" source="n7" target="n11">
+    <edge id="e9" source="n7" target="n10">
       <data key="d10">
         <y:PolyLineEdge>
           <y:Path sx="0.0" sy="28.0" tx="0.0" ty="-28.0"/>
@@ -374,7 +358,7 @@ FreeBSD<y:LabelModel>
         </y:PolyLineEdge>
       </data>
     </edge>
-    <edge id="e10" source="n7" target="n10">
+    <edge id="e10" source="n7" target="n9">
       <data key="d10">
         <y:PolyLineEdge>
           <y:Path sx="-134.01566143419018" sy="6.159084623893818" tx="0.0" ty="-29.333162136535975">
@@ -386,18 +370,7 @@ FreeBSD<y:LabelModel>
         </y:PolyLineEdge>
       </data>
     </edge>
-    <edge id="e11" source="n11" target="n9">
-      <data key="d10">
-        <y:PolyLineEdge>
-          <y:Path sx="0.0" sy="28.0" tx="0.0" ty="-28.86721713021484"/>
-          <y:LineStyle color="#000000" type="line" width="1.0"/>
-          <y:Arrows source="none" target="none"/>
-          <y:BendStyle smoothed="false"/>
-        </y:PolyLineEdge>
-      </data>
-    </edge>
-    <edge id="e12" source="n1" target="n12">
-      <data key="d9"/>
+    <edge id="e11" source="n1" target="n11">
       <data key="d10">
         <y:PolyLineEdge>
           <y:Path sx="0.0" sy="20.0" tx="0.0" ty="-28.0"/>
@@ -415,8 +388,7 @@ FreeBSD<y:LabelModel>
         </y:PolyLineEdge>
       </data>
     </edge>
-    <edge id="e13" source="n12" target="n13">
-      <data key="d9"/>
+    <edge id="e12" source="n11" target="n12">
       <data key="d10">
         <y:PolyLineEdge>
           <y:Path sx="0.0" sy="28.0" tx="0.0" ty="-28.0"/>
@@ -426,8 +398,7 @@ FreeBSD<y:LabelModel>
         </y:PolyLineEdge>
       </data>
     </edge>
-    <edge id="e14" source="n13" target="n7">
-      <data key="d9"/>
+    <edge id="e13" source="n12" target="n7">
       <data key="d10">
         <y:PolyLineEdge>
           <y:Path sx="134.00000000000006" sy="0.0" tx="0.0" ty="-28.0">

main.go

@@ -22,15 +22,20 @@ import (
 	"fmt"
 	"os"
 
-	"golang.org/x/net/context"
+	"context"
 
 	"github.com/future-architect/vuls/commands"
-	"github.com/future-architect/vuls/version"
 	"github.com/google/subcommands"
 
 	_ "github.com/mattn/go-sqlite3"
 )
 
+// Version of Vuls
+var version = "0.2.0"
+
+// Revision of Git
+var revision string
+
 func main() {
 	subcommands.Register(subcommands.HelpCommand(), "")
 	subcommands.Register(subcommands.FlagsCommand(), "")
@@ -40,6 +45,7 @@ func main() {
 	subcommands.Register(&commands.ScanCmd{}, "scan")
 	subcommands.Register(&commands.PrepareCmd{}, "prepare")
 	subcommands.Register(&commands.HistoryCmd{}, "history")
+	subcommands.Register(&commands.ReportCmd{}, "report")
 	subcommands.Register(&commands.ConfigtestCmd{}, "configtest")
 
 	var v = flag.Bool("v", false, "Show version")
@@ -47,7 +53,7 @@ func main() {
 	flag.Parse()
 
 	if *v {
-		fmt.Printf("%s %s\n", version.Name, version.Version)
+		fmt.Printf("vuls %s %s\n", version, revision)
 		os.Exit(int(subcommands.ExitSuccess))
 	}
 

models/models.go

@@ -23,15 +23,13 @@ import (
 	"time"
 
 	"github.com/future-architect/vuls/config"
-	"github.com/jinzhu/gorm"
+	"github.com/future-architect/vuls/cveapi"
 	cve "github.com/kotakanbe/go-cve-dictionary/models"
 )
 
 // ScanHistory is the history of Scanning.
 type ScanHistory struct {
-	gorm.Model
 	ScanResults ScanResults
-	ScannedAt   time.Time
 }
 
 // ScanResults is slice of ScanResult.
@@ -55,47 +53,117 @@ func (s ScanResults) Less(i, j int) bool {
 	return s[i].ServerName < s[j].ServerName
 }
 
-// FilterByCvssOver is filter function.
-func (s ScanResults) FilterByCvssOver() (filtered ScanResults) {
-	for _, result := range s {
-		cveInfos := []CveInfo{}
-		for _, cveInfo := range result.KnownCves {
-			if config.Conf.CvssScoreOver < cveInfo.CveDetail.CvssScore(config.Conf.Lang) {
-				cveInfos = append(cveInfos, cveInfo)
-			}
-		}
-		result.KnownCves = cveInfos
-		filtered = append(filtered, result)
-	}
-	return
-}
-
 // ScanResult has the result of scanned CVE information.
 type ScanResult struct {
-	gorm.Model    `json:"-"`
-	ScanHistoryID uint `json:"-"`
+	ScannedAt time.Time
 
+	Lang       string
 	ServerName string // TOML Section key
-	//  Hostname    string
-	Family  string
-	Release string
+	Family     string
+	Release    string
+	Container  Container
+	Platform   Platform
 
-	Container Container
+	// Scanned Vulns via SSH + CPE Vulns
+	ScannedCves []VulnInfo
 
-	Platform Platform
-
-	//  Fqdn        string
-	//  NWLinks     []NWLink
 	KnownCves   []CveInfo
 	UnknownCves []CveInfo
+	IgnoredCves []CveInfo
 
-	Optional [][]interface{} `gorm:"-"`
+	Packages PackageInfoList
+
+	Optional [][]interface{}
+}
+
+// FillCveDetail fetches CVE detailed information from
+// CVE Database, and then set to fields.
+func (r ScanResult) FillCveDetail() (ScanResult, error) {
+	set := map[string]VulnInfo{}
+	var cveIDs []string
+	for _, v := range r.ScannedCves {
+		set[v.CveID] = v
+		cveIDs = append(cveIDs, v.CveID)
+	}
+
+	ds, err := cveapi.CveClient.FetchCveDetails(cveIDs)
+	if err != nil {
+		return r, err
+	}
+
+	icves := config.Conf.Servers[r.ServerName].IgnoreCves
+
+	var known, unknown, ignored CveInfos
+	for _, d := range ds {
+		cinfo := CveInfo{
+			CveDetail: d,
+			VulnInfo:  set[d.CveID],
+		}
+
+		// ignored
+		found := false
+		for _, icve := range icves {
+			if icve == d.CveID {
+				ignored = append(ignored, cinfo)
+				found = true
+				break
+			}
+		}
+		if found {
+			continue
+		}
+
+		// unknown
+		if d.CvssScore(config.Conf.Lang) <= 0 {
+			unknown = append(unknown, cinfo)
+			continue
+		}
+
+		// known
+		known = append(known, cinfo)
+	}
+	sort.Sort(known)
+	sort.Sort(unknown)
+	sort.Sort(ignored)
+	r.KnownCves = known
+	r.UnknownCves = unknown
+	r.IgnoredCves = ignored
+	return r, nil
+}
+
+// FilterByCvssOver is filter function.
+func (r ScanResult) FilterByCvssOver() ScanResult {
+	cveInfos := []CveInfo{}
+	for _, cveInfo := range r.KnownCves {
+		if config.Conf.CvssScoreOver < cveInfo.CveDetail.CvssScore(config.Conf.Lang) {
+			cveInfos = append(cveInfos, cveInfo)
+		}
+	}
+	r.KnownCves = cveInfos
+	return r
+}
+
+// ReportFileName returns the filename on localhost without extention
+func (r ScanResult) ReportFileName() (name string) {
+	if len(r.Container.ContainerID) == 0 {
+		return fmt.Sprintf("%s", r.ServerName)
+	}
+	return fmt.Sprintf("%s@%s", r.Container.Name, r.ServerName)
+}
+
+// ReportKeyName returns the name of key on S3, Azure-Blob without extention
+func (r ScanResult) ReportKeyName() (name string) {
+	timestr := r.ScannedAt.Format(time.RFC3339)
+	if len(r.Container.ContainerID) == 0 {
+		return fmt.Sprintf("%s/%s", timestr, r.ServerName)
+	}
+	return fmt.Sprintf("%s/%s@%s", timestr, r.Container.Name, r.ServerName)
 }
 
 // ServerInfo returns server name one line
 func (r ScanResult) ServerInfo() string {
 	hostinfo := ""
-	if r.Container.ContainerID == "" {
+	if len(r.Container.ContainerID) == 0 {
 		hostinfo = fmt.Sprintf(
 			"%s (%s%s)",
 			r.ServerName,
@@ -118,7 +186,7 @@ func (r ScanResult) ServerInfo() string {
 // ServerInfoTui returns server infromation for TUI sidebar
 func (r ScanResult) ServerInfoTui() string {
 	hostinfo := ""
-	if r.Container.ContainerID == "" {
+	if len(r.Container.ContainerID) == 0 {
 		hostinfo = fmt.Sprintf(
 			"%s (%s%s)",
 			r.ServerName,
@@ -139,15 +207,15 @@ func (r ScanResult) ServerInfoTui() string {
 
 // CveSummary summarize the number of CVEs group by CVSSv2 Severity
 func (r ScanResult) CveSummary() string {
-	var high, middle, low, unknown int
+	var high, medium, low, unknown int
 	cves := append(r.KnownCves, r.UnknownCves...)
 	for _, cveInfo := range cves {
 		score := cveInfo.CveDetail.CvssScore(config.Conf.Lang)
 		switch {
-		case 7.0 < score:
+		case 7.0 <= score:
 			high++
-		case 4.0 < score:
-			middle++
+		case 4.0 <= score:
+			medium++
 		case 0 < score:
 			low++
 		default:
@@ -156,24 +224,73 @@ func (r ScanResult) CveSummary() string {
 	}
 
 	if config.Conf.IgnoreUnscoredCves {
-		return fmt.Sprintf("Total: %d (High:%d Middle:%d Low:%d)",
-			high+middle+low, high, middle, low)
+		return fmt.Sprintf("Total: %d (High:%d Medium:%d Low:%d)",
+			high+medium+low, high, medium, low)
 	}
-	return fmt.Sprintf("Total: %d (High:%d Middle:%d Low:%d ?:%d)",
-		high+middle+low+unknown, high, middle, low, unknown)
+	return fmt.Sprintf("Total: %d (High:%d Medium:%d Low:%d ?:%d)",
+		high+medium+low+unknown, high, medium, low, unknown)
+}
+
+// AllCves returns Known and Unknown CVEs
+func (r ScanResult) AllCves() []CveInfo {
+	return append(r.KnownCves, r.UnknownCves...)
 }
 
 // NWLink has network link information.
 type NWLink struct {
-	gorm.Model   `json:"-"`
-	ScanResultID uint `json:"-"`
-
 	IPAddress string
 	Netmask   string
 	DevName   string
 	LinkState string
 }
 
+// VulnInfos is VulnInfo list, getter/setter, sortable methods.
+type VulnInfos []VulnInfo
+
+// VulnInfo holds a vulnerability information and unsecure packages
+type VulnInfo struct {
+	CveID            string
+	Packages         PackageInfoList
+	DistroAdvisories []DistroAdvisory // for Aamazon, RHEL, FreeBSD
+	CpeNames         []string
+}
+
+// FindByCveID find by CVEID
+func (s VulnInfos) FindByCveID(cveID string) (VulnInfo, bool) {
+	for _, p := range s {
+		if cveID == p.CveID {
+			return p, true
+		}
+	}
+	return VulnInfo{CveID: cveID}, false
+}
+
+// immutable
+func (s VulnInfos) set(cveID string, v VulnInfo) VulnInfos {
+	for i, p := range s {
+		if cveID == p.CveID {
+			s[i] = v
+			return s
+		}
+	}
+	return append(s, v)
+}
+
+// Len implement Sort Interface
+func (s VulnInfos) Len() int {
+	return len(s)
+}
+
+// Swap implement Sort Interface
+func (s VulnInfos) Swap(i, j int) {
+	s[i], s[j] = s[j], s[i]
+}
+
+// Less implement Sort Interface
+func (s VulnInfos) Less(i, j int) bool {
+	return s[i].CveID < s[j].CveID
+}
+
 // CveInfos is for sorting
 type CveInfos []CveInfo
 
@@ -190,26 +307,13 @@ func (c CveInfos) Less(i, j int) bool {
 	if c[i].CveDetail.CvssScore(lang) == c[j].CveDetail.CvssScore(lang) {
 		return c[i].CveDetail.CveID < c[j].CveDetail.CveID
 	}
-	return c[i].CveDetail.CvssScore(lang) > c[j].CveDetail.CvssScore(lang)
+	return c[j].CveDetail.CvssScore(lang) < c[i].CveDetail.CvssScore(lang)
 }
 
 // CveInfo has Cve Information.
 type CveInfo struct {
-	gorm.Model   `json:"-"`
-	ScanResultID uint `json:"-"`
-
-	CveDetail        cve.CveDetail
-	Packages         []PackageInfo
-	DistroAdvisories []DistroAdvisory
-	CpeNames         []CpeName
-}
-
-// CpeName has CPE name
-type CpeName struct {
-	gorm.Model `json:"-"`
-	CveInfoID  uint `json:"-"`
-
-	Name string
+	CveDetail cve.CveDetail
+	VulnInfo
 }
 
 // PackageInfoList is slice of PackageInfo
@@ -253,6 +357,34 @@ func (ps PackageInfoList) FindByName(name string) (result PackageInfo, found boo
 	return PackageInfo{}, false
 }
 
+// MergeNewVersion merges candidate version information to the receiver struct
+func (ps PackageInfoList) MergeNewVersion(as PackageInfoList) {
+	for _, a := range as {
+		for i, p := range ps {
+			if p.Name == a.Name {
+				ps[i].NewVersion = a.NewVersion
+				ps[i].NewRelease = a.NewRelease
+			}
+		}
+	}
+}
+
+func (ps PackageInfoList) countUpdatablePacks() int {
+	count := 0
+	for _, p := range ps {
+		if len(p.NewVersion) != 0 {
+			count++
+		}
+	}
+	return count
+}
+
+// ToUpdatablePacksSummary returns a summary of updatable packages
+func (ps PackageInfoList) ToUpdatablePacksSummary() string {
+	return fmt.Sprintf("%d updatable packages",
+		ps.countUpdatablePacks())
+}
+
 // Find search PackageInfo by name-version-release
 //  func (ps PackageInfoList) find(nameVersionRelease string) (PackageInfo, bool) {
 //      for _, p := range ps {
@@ -270,17 +402,22 @@ func (ps PackageInfoList) FindByName(name string) (result PackageInfo, found boo
 //      return PackageInfo{}, false
 //  }
 
+// PackageInfosByName implements sort.Interface for []PackageInfo based on
+// the Name field.
+type PackageInfosByName []PackageInfo
+
+func (a PackageInfosByName) Len() int           { return len(a) }
+func (a PackageInfosByName) Swap(i, j int)      { a[i], a[j] = a[j], a[i] }
+func (a PackageInfosByName) Less(i, j int) bool { return a[i].Name < a[j].Name }
+
 // PackageInfo has installed packages.
 type PackageInfo struct {
-	gorm.Model `json:"-"`
-	CveInfoID  uint `json:"-"`
-
-	Name    string
-	Version string
-	Release string
-
+	Name       string
+	Version    string
+	Release    string
 	NewVersion string
 	NewRelease string
+	Repository string
 }
 
 // ToStringCurrentVersion returns package name-version-release
@@ -309,9 +446,6 @@ func (p PackageInfo) ToStringNewVersion() string {
 
 // DistroAdvisory has Amazon Linux, RHEL, FreeBSD Security Advisory information.
 type DistroAdvisory struct {
-	gorm.Model `json:"-"`
-	CveInfoID  uint `json:"-"`
-
 	AdvisoryID string
 	Severity   string
 	Issued     time.Time
@@ -320,18 +454,12 @@ type DistroAdvisory struct {
 
 // Container has Container information
 type Container struct {
-	gorm.Model   `json:"-"`
-	ScanResultID uint `json:"-"`
-
 	ContainerID string
 	Name        string
 }
 
 // Platform has platform information
 type Platform struct {
-	gorm.Model   `json:"-"`
-	ScanResultID uint `json:"-"`
-
 	Name       string // aws or azure or gcp or other...
 	InstanceID string
 }

models/models_test.go

@@ -17,9 +17,14 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 package models
 
-import "testing"
+import (
+	"reflect"
+	"testing"
 
-func TestPackageInfosUniqByName(t *testing.T) {
+	"github.com/k0kubun/pp"
+)
+
+func TestPackageInfoListUniqByName(t *testing.T) {
 	var test = struct {
 		in  PackageInfoList
 		out PackageInfoList
@@ -52,3 +57,81 @@ func TestPackageInfosUniqByName(t *testing.T) {
 		}
 	}
 }
+
+func TestMergeNewVersion(t *testing.T) {
+	var test = struct {
+		a        PackageInfoList
+		b        PackageInfoList
+		expected PackageInfoList
+	}{
+		PackageInfoList{
+			{
+				Name: "hoge",
+			},
+		},
+		PackageInfoList{
+			{
+				Name:       "hoge",
+				NewVersion: "1.0.0",
+				NewRelease: "release1",
+			},
+		},
+		PackageInfoList{
+			{
+				Name:       "hoge",
+				NewVersion: "1.0.0",
+				NewRelease: "release1",
+			},
+		},
+	}
+
+	test.a.MergeNewVersion(test.b)
+	if !reflect.DeepEqual(test.a, test.expected) {
+		e := pp.Sprintf("%v", test.a)
+		a := pp.Sprintf("%v", test.expected)
+		t.Errorf("expected %s, actual %s", e, a)
+	}
+}
+func TestVulnInfosSetGet(t *testing.T) {
+	var test = struct {
+		in  []string
+		out []string
+	}{
+		[]string{
+			"CVE1",
+			"CVE2",
+			"CVE3",
+			"CVE1",
+			"CVE1",
+			"CVE2",
+			"CVE3",
+		},
+		[]string{
+			"CVE1",
+			"CVE2",
+			"CVE3",
+		},
+	}
+
+	//  var ps packageCveInfos
+	var ps VulnInfos
+	for _, cid := range test.in {
+		ps = ps.set(cid, VulnInfo{CveID: cid})
+	}
+
+	if len(test.out) != len(ps) {
+		t.Errorf("length: expected %d, actual %d", len(test.out), len(ps))
+	}
+
+	for i, expectedCid := range test.out {
+		if expectedCid != ps[i].CveID {
+			t.Errorf("expected %s, actual %s", expectedCid, ps[i].CveID)
+		}
+	}
+	for _, cid := range test.in {
+		p, _ := ps.FindByCveID(cid)
+		if p.CveID != cid {
+			t.Errorf("expected %s, actual %s", cid, p.CveID)
+		}
+	}
+}

report/azureblob.go

@@ -20,6 +20,7 @@ package report
 import (
 	"bytes"
 	"encoding/json"
+	"encoding/xml"
 	"fmt"
 	"time"
 
@@ -27,12 +28,76 @@ import (
 
 	c "github.com/future-architect/vuls/config"
 	"github.com/future-architect/vuls/models"
-	"github.com/future-architect/vuls/util"
 )
 
 // AzureBlobWriter writes results to AzureBlob
 type AzureBlobWriter struct{}
 
+// Write results to Azure Blob storage
+func (w AzureBlobWriter) Write(rs ...models.ScanResult) (err error) {
+	if len(rs) == 0 {
+		return nil
+	}
+
+	cli, err := getBlobClient()
+	if err != nil {
+		return err
+	}
+
+	if c.Conf.FormatOneLineText {
+		timestr := rs[0].ScannedAt.Format(time.RFC3339)
+		k := fmt.Sprintf(timestr + "/summary.txt")
+		text := toOneLineSummary(rs...)
+		b := []byte(text)
+		if err := createBlockBlob(cli, k, b); err != nil {
+			return err
+		}
+	}
+
+	for _, r := range rs {
+		key := r.ReportKeyName()
+		if c.Conf.FormatJSON {
+			k := key + ".json"
+			var b []byte
+			if b, err = json.Marshal(r); err != nil {
+				return fmt.Errorf("Failed to Marshal to JSON: %s", err)
+			}
+			if err := createBlockBlob(cli, k, b); err != nil {
+				return err
+			}
+		}
+
+		if c.Conf.FormatShortText {
+			k := key + "_short.txt"
+			b := []byte(toShortPlainText(r))
+			if err := createBlockBlob(cli, k, b); err != nil {
+				return err
+			}
+		}
+
+		if c.Conf.FormatFullText {
+			k := key + "_full.txt"
+			b := []byte(toFullPlainText(r))
+			if err := createBlockBlob(cli, k, b); err != nil {
+				return err
+			}
+		}
+
+		if c.Conf.FormatXML {
+			k := key + ".xml"
+			var b []byte
+			if b, err = xml.Marshal(r); err != nil {
+				return fmt.Errorf("Failed to Marshal to XML: %s", err)
+			}
+			allBytes := bytes.Join([][]byte{[]byte(xml.Header + vulsOpenTag), b, []byte(vulsCloseTag)}, []byte{})
+			if err := createBlockBlob(cli, k, allBytes); err != nil {
+				return err
+			}
+		}
+	}
+	return
+}
+
 // CheckIfAzureContainerExists check the existence of Azure storage container
 func CheckIfAzureContainerExists() error {
 	cli, err := getBlobClient()
@@ -57,84 +122,24 @@ func getBlobClient() (storage.BlobStorageClient, error) {
 	return api.GetBlobService(), nil
 }
 
-// Write results to Azure Blob storage
-func (w AzureBlobWriter) Write(scanResults []models.ScanResult) (err error) {
-	reqChan := make(chan models.ScanResult, len(scanResults))
-	resChan := make(chan bool)
-	errChan := make(chan error, len(scanResults))
-	defer close(resChan)
-	defer close(errChan)
-	defer close(reqChan)
-
-	timeout := time.After(10 * 60 * time.Second)
-	concurrency := 10
-	tasks := util.GenWorkers(concurrency)
-
-	go func() {
-		for _, r := range scanResults {
-			reqChan <- r
-		}
-	}()
-
-	for range scanResults {
-		tasks <- func() {
-			select {
-			case sresult := <-reqChan:
-				func(r models.ScanResult) {
-					err := w.upload(r)
-					if err != nil {
-						errChan <- err
-					}
-					resChan <- true
-				}(sresult)
-			}
+func createBlockBlob(cli storage.BlobStorageClient, k string, b []byte) error {
+	var err error
+	if c.Conf.GZIP {
+		if b, err = gz(b); err != nil {
+			return err
 		}
+		k = k + ".gz"
 	}
 
-	errs := []error{}
-	for i := 0; i < len(scanResults); i++ {
-		select {
-		case <-resChan:
-		case err := <-errChan:
-			errs = append(errs, err)
-		case <-timeout:
-			errs = append(errs, fmt.Errorf("Timeout while uploading to azure Blob"))
-		}
-	}
-
-	if 0 < len(errs) {
-		return fmt.Errorf("Failed to upload json to Azure Blob: %v", errs)
+	if err := cli.CreateBlockBlobFromReader(
+		c.Conf.AzureContainer,
+		k,
+		uint64(len(b)),
+		bytes.NewReader(b),
+		map[string]string{},
+	); err != nil {
+		return fmt.Errorf("Failed to upload data to %s/%s, %s",
+			c.Conf.AzureContainer, k, err)
 	}
 	return nil
 }
-
-func (w AzureBlobWriter) upload(res models.ScanResult) (err error) {
-	cli, err := getBlobClient()
-	if err != nil {
-		return err
-	}
-	timestr := time.Now().Format("20060102_1504")
-	name := ""
-	if res.Container.ContainerID == "" {
-		name = fmt.Sprintf("%s/%s.json", timestr, res.ServerName)
-	} else {
-		name = fmt.Sprintf("%s/%s_%s.json", timestr, res.ServerName, res.Container.Name)
-	}
-
-	jsonBytes, err := json.Marshal(res)
-	if err != nil {
-		return fmt.Errorf("Failed to Marshal to JSON: %s", err)
-	}
-
-	if err = cli.CreateBlockBlobFromReader(
-		c.Conf.AzureContainer,
-		name,
-		uint64(len(jsonBytes)),
-		bytes.NewReader(jsonBytes),
-		map[string]string{},
-	); err != nil {
-		return fmt.Errorf("%s/%s, %s",
-			c.Conf.AzureContainer, name, err)
-	}
-	return
-}

report/email.go

@@ -0,0 +1,84 @@
+/* Vuls - Vulnerability Scanner
+Copyright (C) 2016  Future Architect, Inc. Japan.
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+package report
+
+import (
+	"fmt"
+	"net"
+	"net/mail"
+	"net/smtp"
+	"strings"
+	"time"
+
+	"github.com/future-architect/vuls/config"
+	"github.com/future-architect/vuls/models"
+)
+
+// EMailWriter send mail
+type EMailWriter struct{}
+
+func (w EMailWriter) Write(rs ...models.ScanResult) (err error) {
+	conf := config.Conf
+	to := strings.Join(conf.EMail.To[:], ", ")
+	cc := strings.Join(conf.EMail.Cc[:], ", ")
+	mailAddresses := append(conf.EMail.To, conf.EMail.Cc...)
+	if _, err := mail.ParseAddressList(strings.Join(mailAddresses[:], ", ")); err != nil {
+		return fmt.Errorf("Failed to parse email addresses: %s", err)
+	}
+
+	for _, r := range rs {
+		subject := fmt.Sprintf("%s%s %s",
+			conf.EMail.SubjectPrefix,
+			r.ServerInfo(),
+			r.CveSummary(),
+		)
+
+		headers := make(map[string]string)
+		headers["From"] = conf.EMail.From
+		headers["To"] = to
+		headers["Cc"] = cc
+		headers["Subject"] = subject
+		headers["Date"] = time.Now().Format(time.RFC1123Z)
+		headers["Content-Type"] = "text/plain; charset=utf-8"
+
+		var message string
+		for k, v := range headers {
+			message += fmt.Sprintf("%s: %s\r\n", k, v)
+		}
+		message += "\r\n" + toFullPlainText(r)
+
+		smtpServer := net.JoinHostPort(conf.EMail.SMTPAddr, conf.EMail.SMTPPort)
+		err = smtp.SendMail(
+			smtpServer,
+			smtp.PlainAuth(
+				"",
+				conf.EMail.User,
+				conf.EMail.Password,
+				conf.EMail.SMTPAddr,
+			),
+			conf.EMail.From,
+			conf.EMail.To,
+			[]byte(message),
+		)
+
+		if err != nil {
+			return fmt.Errorf("Failed to send emails: %s", err)
+		}
+	}
+	return nil
+}

report/json.go

@@ -1,62 +0,0 @@
-/* Vuls - Vulnerability Scanner
-Copyright (C) 2016  Future Architect, Inc. Japan.
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-package report
-
-import (
-	"encoding/json"
-	"fmt"
-	"io/ioutil"
-	"path/filepath"
-
-	"github.com/future-architect/vuls/models"
-)
-
-// JSONWriter writes results to file.
-type JSONWriter struct{}
-
-func (w JSONWriter) Write(scanResults []models.ScanResult) (err error) {
-
-	path, err := ensureResultDir()
-
-	var jsonBytes []byte
-	if jsonBytes, err = json.Marshal(scanResults); err != nil {
-		return fmt.Errorf("Failed to Marshal to JSON: %s", err)
-	}
-	all := filepath.Join(path, "all.json")
-	if err := ioutil.WriteFile(all, jsonBytes, 0644); err != nil {
-		return fmt.Errorf("Failed to write JSON. path: %s, err: %s", all, err)
-	}
-
-	for _, r := range scanResults {
-		jsonPath := ""
-		if r.Container.ContainerID == "" {
-			jsonPath = filepath.Join(path, fmt.Sprintf("%s.json", r.ServerName))
-		} else {
-			jsonPath = filepath.Join(path,
-				fmt.Sprintf("%s_%s.json", r.ServerName, r.Container.Name))
-		}
-
-		if jsonBytes, err = json.Marshal(r); err != nil {
-			return fmt.Errorf("Failed to Marshal to JSON: %s", err)
-		}
-		if err := ioutil.WriteFile(jsonPath, jsonBytes, 0644); err != nil {
-			return fmt.Errorf("Failed to write JSON. path: %s, err: %s", jsonPath, err)
-		}
-	}
-	return nil
-}

report/localfile.go

@@ -0,0 +1,111 @@
+/* Vuls - Vulnerability Scanner
+Copyright (C) 2016  Future Architect, Inc. Japan.
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+package report
+
+import (
+	"bytes"
+	"encoding/json"
+	"encoding/xml"
+	"fmt"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+
+	c "github.com/future-architect/vuls/config"
+	"github.com/future-architect/vuls/models"
+)
+
+// LocalFileWriter writes results to a local file.
+type LocalFileWriter struct {
+	CurrentDir string
+}
+
+func (w LocalFileWriter) Write(rs ...models.ScanResult) (err error) {
+	if c.Conf.FormatOneLineText {
+		path := filepath.Join(w.CurrentDir, "summary.txt")
+		text := toOneLineSummary(rs...)
+		if err := writeFile(path, []byte(text), 0600); err != nil {
+			return fmt.Errorf(
+				"Failed to write to file. path: %s, err: %s",
+				path, err)
+		}
+	}
+
+	for _, r := range rs {
+		path := filepath.Join(w.CurrentDir, r.ReportFileName())
+
+		if c.Conf.FormatJSON {
+			p := path + ".json"
+			var b []byte
+			if b, err = json.Marshal(r); err != nil {
+				return fmt.Errorf("Failed to Marshal to JSON: %s", err)
+			}
+			if err := writeFile(p, b, 0600); err != nil {
+				return fmt.Errorf("Failed to write JSON. path: %s, err: %s", p, err)
+			}
+		}
+
+		if c.Conf.FormatShortText {
+			p := path + "_short.txt"
+			if err := writeFile(
+				p, []byte(toShortPlainText(r)), 0600); err != nil {
+				return fmt.Errorf(
+					"Failed to write text files. path: %s, err: %s", p, err)
+			}
+		}
+
+		if c.Conf.FormatFullText {
+			p := path + "_full.txt"
+			if err := writeFile(
+				p, []byte(toFullPlainText(r)), 0600); err != nil {
+				return fmt.Errorf(
+					"Failed to write text files. path: %s, err: %s", p, err)
+			}
+		}
+
+		if c.Conf.FormatXML {
+			p := path + ".xml"
+			var b []byte
+			if b, err = xml.Marshal(r); err != nil {
+				return fmt.Errorf("Failed to Marshal to XML: %s", err)
+			}
+			allBytes := bytes.Join([][]byte{[]byte(xml.Header + vulsOpenTag), b, []byte(vulsCloseTag)}, []byte{})
+			if err := writeFile(p, allBytes, 0600); err != nil {
+				return fmt.Errorf("Failed to write XML. path: %s, err: %s", p, err)
+			}
+		}
+	}
+	return nil
+}
+
+func writeFile(path string, data []byte, perm os.FileMode) error {
+	var err error
+	if c.Conf.GZIP {
+		if data, err = gz(data); err != nil {
+			return err
+		}
+		path = path + ".gz"
+	}
+
+	if err := ioutil.WriteFile(
+		path, []byte(data), perm); err != nil {
+		return err
+	}
+
+	return nil
+}

report/logrus.go

@@ -1,56 +0,0 @@
-/* Vuls - Vulnerability Scanner
-Copyright (C) 2016  Future Architect, Inc. Japan.
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-package report
-
-import (
-	"os"
-	"path/filepath"
-	"runtime"
-
-	"github.com/Sirupsen/logrus"
-	"github.com/future-architect/vuls/models"
-	formatter "github.com/kotakanbe/logrus-prefixed-formatter"
-)
-
-// LogrusWriter write to logfile
-type LogrusWriter struct {
-}
-
-func (w LogrusWriter) Write(scanResults []models.ScanResult) error {
-	path := "/var/log/vuls/report.log"
-	if runtime.GOOS == "windows" {
-		path = filepath.Join(os.Getenv("APPDATA"), "vuls", "report.log")
-	}
-	f, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
-	if err != nil {
-		return err
-	}
-	log := logrus.New()
-	log.Formatter = &formatter.TextFormatter{}
-	log.Out = f
-	log.Level = logrus.InfoLevel
-
-	for _, s := range scanResults {
-		text, err := toPlainText(s)
-		if err != nil {
-			return err
-		}
-		log.Infof(text)
-	}
-	return nil
-}

report/mail.go

@@ -1,70 +0,0 @@
-/* Vuls - Vulnerability Scanner
-Copyright (C) 2016  Future Architect, Inc. Japan.
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-package report
-
-import (
-	"crypto/tls"
-	"fmt"
-	"strconv"
-
-	"github.com/future-architect/vuls/config"
-	"github.com/future-architect/vuls/models"
-	"gopkg.in/gomail.v2"
-)
-
-// MailWriter send mail
-type MailWriter struct{}
-
-func (w MailWriter) Write(scanResults []models.ScanResult) (err error) {
-	conf := config.Conf
-	for _, s := range scanResults {
-		m := gomail.NewMessage()
-		m.SetHeader("From", conf.Mail.From)
-		m.SetHeader("To", conf.Mail.To...)
-		m.SetHeader("Cc", conf.Mail.Cc...)
-
-		subject := fmt.Sprintf("%s%s %s",
-			conf.Mail.SubjectPrefix,
-			s.ServerInfo(),
-			s.CveSummary(),
-		)
-		m.SetHeader("Subject", subject)
-
-		var body string
-		if body, err = toPlainText(s); err != nil {
-			return err
-		}
-		m.SetBody("text/plain", body)
-		port, _ := strconv.Atoi(conf.Mail.SMTPPort)
-		d := gomail.NewPlainDialer(
-			conf.Mail.SMTPAddr,
-			port,
-			conf.Mail.User,
-			conf.Mail.Password,
-		)
-
-		d.TLSConfig = &tls.Config{
-			InsecureSkipVerify: true,
-		}
-
-		if err := d.DialAndSend(m); err != nil {
-			panic(err)
-		}
-	}
-	return nil
-}

report/s3.go

@@ -20,6 +20,7 @@ package report
 import (
 	"bytes"
 	"encoding/json"
+	"encoding/xml"
 	"fmt"
 	"time"
 
@@ -32,6 +33,78 @@ import (
 	"github.com/future-architect/vuls/models"
 )
 
+// S3Writer writes results to S3
+type S3Writer struct{}
+
+func getS3() *s3.S3 {
+	return s3.New(session.New(&aws.Config{
+		Region:      aws.String(c.Conf.AwsRegion),
+		Credentials: credentials.NewSharedCredentials("", c.Conf.AwsProfile),
+	}))
+}
+
+// Write results to S3
+// http://docs.aws.amazon.com/sdk-for-go/latest/v1/developerguide/common-examples.title.html
+func (w S3Writer) Write(rs ...models.ScanResult) (err error) {
+	if len(rs) == 0 {
+		return nil
+	}
+
+	svc := getS3()
+
+	if c.Conf.FormatOneLineText {
+		timestr := rs[0].ScannedAt.Format(time.RFC3339)
+		k := fmt.Sprintf(timestr + "/summary.txt")
+		text := toOneLineSummary(rs...)
+		if err := putObject(svc, k, []byte(text)); err != nil {
+			return err
+		}
+	}
+
+	for _, r := range rs {
+		key := r.ReportKeyName()
+		if c.Conf.FormatJSON {
+			k := key + ".json"
+			var b []byte
+			if b, err = json.Marshal(r); err != nil {
+				return fmt.Errorf("Failed to Marshal to JSON: %s", err)
+			}
+			if err := putObject(svc, k, b); err != nil {
+				return err
+			}
+		}
+
+		if c.Conf.FormatShortText {
+			k := key + "_short.txt"
+			text := toShortPlainText(r)
+			if err := putObject(svc, k, []byte(text)); err != nil {
+				return err
+			}
+		}
+
+		if c.Conf.FormatFullText {
+			k := key + "_full.txt"
+			text := toFullPlainText(r)
+			if err := putObject(svc, k, []byte(text)); err != nil {
+				return err
+			}
+		}
+
+		if c.Conf.FormatXML {
+			k := key + ".xml"
+			var b []byte
+			if b, err = xml.Marshal(r); err != nil {
+				return fmt.Errorf("Failed to Marshal to XML: %s", err)
+			}
+			allBytes := bytes.Join([][]byte{[]byte(xml.Header + vulsOpenTag), b, []byte(vulsCloseTag)}, []byte{})
+			if err := putObject(svc, k, allBytes); err != nil {
+				return err
+			}
+		}
+	}
+	return nil
+}
+
 // CheckIfBucketExists check the existence of S3 bucket
 func CheckIfBucketExists() error {
 	svc := getS3()
@@ -57,56 +130,22 @@ func CheckIfBucketExists() error {
 	return nil
 }
 
-// S3Writer writes results to S3
-type S3Writer struct{}
-
-func getS3() *s3.S3 {
-	return s3.New(session.New(&aws.Config{
-		Region:      aws.String(c.Conf.AwsRegion),
-		Credentials: credentials.NewSharedCredentials("", c.Conf.AwsProfile),
-	}))
-}
-
-// Write results to S3
-func (w S3Writer) Write(scanResults []models.ScanResult) (err error) {
-
-	var jsonBytes []byte
-	if jsonBytes, err = json.Marshal(scanResults); err != nil {
-		return fmt.Errorf("Failed to Marshal to JSON: %s", err)
+func putObject(svc *s3.S3, k string, b []byte) error {
+	var err error
+	if c.Conf.GZIP {
+		if b, err = gz(b); err != nil {
+			return err
+		}
+		k = k + ".gz"
 	}
 
-	// http://docs.aws.amazon.com/sdk-for-go/latest/v1/developerguide/common-examples.title.html
-	svc := getS3()
-	timestr := time.Now().Format("20060102_1504")
-	key := fmt.Sprintf("%s/%s", timestr, "all.json")
-	_, err = svc.PutObject(&s3.PutObjectInput{
+	if _, err := svc.PutObject(&s3.PutObjectInput{
 		Bucket: &c.Conf.S3Bucket,
-		Key:    &key,
-		Body:   bytes.NewReader(jsonBytes),
-	})
-	if err != nil {
-		return fmt.Errorf("Failed to upload data to %s/%s, %s", c.Conf.S3Bucket, key, err)
-	}
-
-	for _, r := range scanResults {
-		key := ""
-		if r.Container.ContainerID == "" {
-			key = fmt.Sprintf("%s/%s.json", timestr, r.ServerName)
-		} else {
-			key = fmt.Sprintf("%s/%s_%s.json", timestr, r.ServerName, r.Container.Name)
-		}
-
-		if jsonBytes, err = json.Marshal(r); err != nil {
-			return fmt.Errorf("Failed to Marshal to JSON: %s", err)
-		}
-		_, err = svc.PutObject(&s3.PutObjectInput{
-			Bucket: &c.Conf.S3Bucket,
-			Key:    &key,
-			Body:   bytes.NewReader(jsonBytes),
-		})
-		if err != nil {
-			return fmt.Errorf("Failed to upload data to %s/%s, %s", c.Conf.S3Bucket, key, err)
-		}
+		Key:    &k,
+		Body:   bytes.NewReader(b),
+	}); err != nil {
+		return fmt.Errorf("Failed to upload data to %s/%s, %s",
+			c.Conf.S3Bucket, k, err)
 	}
 	return nil
 }

report/slack.go

@@ -56,37 +56,36 @@ type message struct {
 // SlackWriter send report to slack
 type SlackWriter struct{}
 
-func (w SlackWriter) Write(scanResults []models.ScanResult) error {
+func (w SlackWriter) Write(rs ...models.ScanResult) error {
 	conf := config.Conf.Slack
-	for _, s := range scanResults {
+	channel := conf.Channel
 
-		channel := conf.Channel
+	for _, r := range rs {
 		if channel == "${servername}" {
-			channel = fmt.Sprintf("#%s", s.ServerName)
+			channel = fmt.Sprintf("#%s", r.ServerName)
 		}
 
 		msg := message{
-			Text:        msgText(s),
+			Text:        msgText(r),
 			Username:    conf.AuthUser,
 			IconEmoji:   conf.IconEmoji,
 			Channel:     channel,
-			Attachments: toSlackAttachments(s),
+			Attachments: toSlackAttachments(r),
 		}
 
 		bytes, _ := json.Marshal(msg)
 		jsonBody := string(bytes)
 		f := func() (err error) {
-			resp, body, errs := gorequest.New().Proxy(config.Conf.HTTPProxy).Post(conf.HookURL).
-				Send(string(jsonBody)).End()
-			if resp.StatusCode != 200 {
-				log.Errorf("Resonse body: %s", body)
-				if len(errs) > 0 {
-					return errs[0]
-				}
+			resp, body, errs := gorequest.New().Proxy(config.Conf.HTTPProxy).Post(conf.HookURL).Send(string(jsonBody)).End()
+			if 0 < len(errs) || resp == nil || resp.StatusCode != 200 {
+				return fmt.Errorf(
+					"HTTP POST error: %v, url: %s, resp: %v, body: %s",
+					errs, conf.HookURL, resp, body)
 			}
 			return nil
 		}
 		notify := func(err error, t time.Duration) {
+			log.Warn("Error %s", err)
 			log.Warn("Retrying in ", t)
 		}
 		if err := backoff.RetryNotify(f, backoff.NewExponentialBackOff(), notify); err != nil {
@@ -97,7 +96,6 @@ func (w SlackWriter) Write(scanResults []models.ScanResult) error {
 }
 
 func msgText(r models.ScanResult) string {
-
 	notifyUsers := ""
 	if 0 < len(r.KnownCves) || 0 < len(r.UnknownCves) {
 		notifyUsers = getNotifyUsers(config.Conf.Slack.NotifyUsers)
@@ -108,7 +106,6 @@ func msgText(r models.ScanResult) string {
 }
 
 func toSlackAttachments(scanResult models.ScanResult) (attaches []*attachment) {
-
 	cves := scanResult.KnownCves
 	if !config.Conf.IgnoreUnscoredCves {
 		cves = append(cves, scanResult.UnknownCves...)
@@ -121,8 +118,8 @@ func toSlackAttachments(scanResult models.ScanResult) (attaches []*attachment) {
 		for _, p := range cveInfo.Packages {
 			curentPackages = append(curentPackages, p.ToStringCurrentVersion())
 		}
-		for _, cpename := range cveInfo.CpeNames {
-			curentPackages = append(curentPackages, cpename.Name)
+		for _, n := range cveInfo.CpeNames {
+			curentPackages = append(curentPackages, n)
 		}
 
 		newPackages := []string{}
@@ -205,6 +202,17 @@ func attachmentText(cveInfo models.CveInfo, osFamily string) string {
 
 func links(cveInfo models.CveInfo, osFamily string) string {
 	links := []string{}
+
+	cweID := cveInfo.CveDetail.CweID()
+	if 0 < len(cweID) {
+		links = append(links, fmt.Sprintf("<%s|%s>",
+			cweURL(cweID), cweID))
+		if config.Conf.Lang == "ja" {
+			links = append(links, fmt.Sprintf("<%s|%s(JVN)>",
+				cweJvnURL(cweID), cweID))
+		}
+	}
+
 	cveID := cveInfo.CveDetail.CveID
 	if config.Conf.Lang == "ja" && 0 < len(cveInfo.CveDetail.Jvn.Link()) {
 		jvn := fmt.Sprintf("<%s|JVN>", cveInfo.CveDetail.Jvn.Link())

report/stdout.go

@@ -20,19 +20,40 @@ package report
 import (
 	"fmt"
 
+	c "github.com/future-architect/vuls/config"
 	"github.com/future-architect/vuls/models"
 )
 
 // StdoutWriter write to stdout
 type StdoutWriter struct{}
 
-func (w StdoutWriter) Write(scanResults []models.ScanResult) error {
-	for _, s := range scanResults {
-		text, err := toPlainText(s)
-		if err != nil {
-			return err
+// WriteScanSummary prints Scan summary at the end of scan
+func (w StdoutWriter) WriteScanSummary(rs ...models.ScanResult) {
+	fmt.Printf("\n\n")
+	fmt.Printf("Scan Summary\n")
+	fmt.Printf("============\n")
+	fmt.Printf("%s\n", toScanSummary(rs...))
+}
+
+func (w StdoutWriter) Write(rs ...models.ScanResult) error {
+	if c.Conf.FormatOneLineText {
+		fmt.Print("\n\n")
+		fmt.Println("One Line Summary")
+		fmt.Println("================")
+		fmt.Println(toOneLineSummary(rs...))
+		fmt.Print("\n")
+	}
+
+	if c.Conf.FormatShortText {
+		for _, r := range rs {
+			fmt.Println(toShortPlainText(r))
+		}
+	}
+
+	if c.Conf.FormatFullText {
+		for _, r := range rs {
+			fmt.Println(toFullPlainText(r))
 		}
-		fmt.Println(text)
 	}
 	return nil
 }

report/textfile.go

@@ -1,63 +0,0 @@
-/* Vuls - Vulnerability Scanner
-Copyright (C) 2016  Future Architect, Inc. Japan.
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-package report
-
-import (
-	"fmt"
-	"io/ioutil"
-	"path/filepath"
-	"strings"
-
-	"github.com/future-architect/vuls/models"
-)
-
-// TextFileWriter writes results to file.
-type TextFileWriter struct{}
-
-func (w TextFileWriter) Write(scanResults []models.ScanResult) (err error) {
-
-	path, err := ensureResultDir()
-
-	all := []string{}
-	for _, r := range scanResults {
-		textFilePath := ""
-		if r.Container.ContainerID == "" {
-			textFilePath = filepath.Join(path, fmt.Sprintf("%s.txt", r.ServerName))
-		} else {
-			textFilePath = filepath.Join(path,
-				fmt.Sprintf("%s_%s.txt", r.ServerName, r.Container.Name))
-		}
-		text, err := toPlainText(r)
-		if err != nil {
-			return err
-		}
-		all = append(all, text)
-		b := []byte(text)
-		if err := ioutil.WriteFile(textFilePath, b, 0644); err != nil {
-			return fmt.Errorf("Failed to write text files. path: %s, err: %s", textFilePath, err)
-		}
-	}
-
-	text := strings.Join(all, "\n\n")
-	b := []byte(text)
-	allPath := filepath.Join(path, "all.txt")
-	if err := ioutil.WriteFile(allPath, b, 0644); err != nil {
-		return fmt.Errorf("Failed to write text files. path: %s, err: %s", allPath, err)
-	}
-	return nil
-}

report/tui.go

@@ -20,13 +20,13 @@ package report
 import (
 	"bytes"
 	"fmt"
+	"os"
 	"strings"
 	"text/template"
 	"time"
 
 	log "github.com/Sirupsen/logrus"
 	"github.com/future-architect/vuls/config"
-	"github.com/future-architect/vuls/db"
 	"github.com/future-architect/vuls/models"
 	"github.com/google/subcommands"
 	"github.com/gosuri/uitable"
@@ -40,45 +40,33 @@ var currentCveInfo int
 var currentDetailLimitY int
 
 // RunTui execute main logic
-func RunTui(historyID string) subcommands.ExitStatus {
-	var err error
-	scanHistory, err = selectScanHistory(historyID)
-	if err != nil {
-		log.Fatal(err)
-		return subcommands.ExitFailure
-	}
+func RunTui(history models.ScanHistory) subcommands.ExitStatus {
+	scanHistory = history
 
-	g := gocui.NewGui()
-	if err := g.Init(); err != nil {
-		log.Panicln(err)
+	g, err := gocui.NewGui(gocui.OutputNormal)
+	if err != nil {
+		log.Errorf("%s", err)
+		return subcommands.ExitFailure
 	}
 	defer g.Close()
 
-	g.SetLayout(layout)
+	g.SetManagerFunc(layout)
 	if err := keybindings(g); err != nil {
-		log.Panicln(err)
+		log.Errorf("%s", err)
+		return subcommands.ExitFailure
 	}
 	g.SelBgColor = gocui.ColorGreen
 	g.SelFgColor = gocui.ColorBlack
 	g.Cursor = true
 
-	if err := g.MainLoop(); err != nil && err != gocui.ErrQuit {
-		log.Panicln(err)
-		return subcommands.ExitFailure
+	if err := g.MainLoop(); err != nil {
+		g.Close()
+		log.Errorf("%s", err)
+		os.Exit(1)
 	}
-
 	return subcommands.ExitSuccess
 }
 
-func selectScanHistory(historyID string) (latest models.ScanHistory, err error) {
-	if err := db.OpenDB(); err != nil {
-		return latest, fmt.Errorf(
-			"Failed to open DB. datafile: %s, err: %s", config.Conf.DBPath, err)
-	}
-	latest, err = db.SelectScanHistory(historyID)
-	return
-}
-
 func keybindings(g *gocui.Gui) (err error) {
 	errs := []error{}
 
@@ -163,35 +151,41 @@ func keybindings(g *gocui.Gui) (err error) {
 }
 
 func nextView(g *gocui.Gui, v *gocui.View) error {
+	var err error
+
 	if v == nil {
-		return g.SetCurrentView("side")
+		_, err = g.SetCurrentView("side")
 	}
 	switch v.Name() {
 	case "side":
-		return g.SetCurrentView("summary")
+		_, err = g.SetCurrentView("summary")
 	case "summary":
-		return g.SetCurrentView("detail")
+		_, err = g.SetCurrentView("detail")
 	case "detail":
-		return g.SetCurrentView("side")
+		_, err = g.SetCurrentView("side")
 	default:
-		return g.SetCurrentView("summary")
+		_, err = g.SetCurrentView("summary")
 	}
+	return err
 }
 
 func previousView(g *gocui.Gui, v *gocui.View) error {
+	var err error
+
 	if v == nil {
-		return g.SetCurrentView("side")
+		_, err = g.SetCurrentView("side")
 	}
 	switch v.Name() {
 	case "side":
-		return g.SetCurrentView("side")
+		_, err = g.SetCurrentView("side")
 	case "summary":
-		return g.SetCurrentView("side")
+		_, err = g.SetCurrentView("side")
 	case "detail":
-		return g.SetCurrentView("summary")
+		_, err = g.SetCurrentView("summary")
 	default:
-		return g.SetCurrentView("side")
+		_, err = g.SetCurrentView("side")
 	}
+	return err
 }
 
 func movable(v *gocui.View, nextY int) (ok bool, yLimit int) {
@@ -203,7 +197,7 @@ func movable(v *gocui.View, nextY int) (ok bool, yLimit int) {
 		}
 		return true, yLimit
 	case "summary":
-		yLimit = len(currentScanResult.KnownCves) - 1
+		yLimit = len(currentScanResult.AllCves()) - 1
 		if yLimit < nextY {
 			return false, yLimit
 		}
@@ -332,7 +326,7 @@ func cursorUp(g *gocui.Gui, v *gocui.View) error {
 	if v != nil {
 		ox, oy := v.Origin()
 		cx, cy := v.Cursor()
-		if err := v.SetCursor(cx, cy-1); err != nil && oy > 0 {
+		if err := v.SetCursor(cx, cy-1); err != nil && 0 < oy {
 			if err := v.SetOrigin(ox, oy-1); err != nil {
 				return err
 			}
@@ -360,7 +354,7 @@ func cursorPageUp(g *gocui.Gui, v *gocui.View) error {
 func previousSummary(g *gocui.Gui, v *gocui.View) error {
 	if v != nil {
 		// cursor to summary
-		if err := g.SetCurrentView("summary"); err != nil {
+		if _, err := g.SetCurrentView("summary"); err != nil {
 			return err
 		}
 		// move next line
@@ -368,7 +362,7 @@ func previousSummary(g *gocui.Gui, v *gocui.View) error {
 			return err
 		}
 		// cursor to detail
-		if err := g.SetCurrentView("detail"); err != nil {
+		if _, err := g.SetCurrentView("detail"); err != nil {
 			return err
 		}
 	}
@@ -378,7 +372,7 @@ func previousSummary(g *gocui.Gui, v *gocui.View) error {
 func nextSummary(g *gocui.Gui, v *gocui.View) error {
 	if v != nil {
 		// cursor to summary
-		if err := g.SetCurrentView("summary"); err != nil {
+		if _, err := g.SetCurrentView("summary"); err != nil {
 			return err
 		}
 		// move next line
@@ -386,7 +380,7 @@ func nextSummary(g *gocui.Gui, v *gocui.View) error {
 			return err
 		}
 		// cursor to detail
-		if err := g.SetCurrentView("detail"); err != nil {
+		if _, err := g.SetCurrentView("detail"); err != nil {
 			return err
 		}
 	}
@@ -451,7 +445,7 @@ func getLine(g *gocui.Gui, v *gocui.View) error {
 			return err
 		}
 		fmt.Fprintln(v, l)
-		if err := g.SetCurrentView("msg"); err != nil {
+		if _, err := g.SetCurrentView("msg"); err != nil {
 			return err
 		}
 	}
@@ -473,7 +467,7 @@ func showMsg(g *gocui.Gui, v *gocui.View) error {
 			return err
 		}
 		fmt.Fprintln(v, l)
-		if err := g.SetCurrentView("msg"); err != nil {
+		if _, err := g.SetCurrentView("msg"); err != nil {
 			return err
 		}
 	}
@@ -484,7 +478,7 @@ func delMsg(g *gocui.Gui, v *gocui.View) error {
 	if err := g.DeleteView("msg"); err != nil {
 		return err
 	}
-	if err := g.SetCurrentView("summary"); err != nil {
+	if _, err := g.SetCurrentView("summary"); err != nil {
 		return err
 	}
 	return nil
@@ -518,8 +512,11 @@ func setSideLayout(g *gocui.Gui) error {
 		for _, result := range scanHistory.ScanResults {
 			fmt.Fprintln(v, result.ServerInfoTui())
 		}
+		if len(scanHistory.ScanResults) == 0 {
+			return fmt.Errorf("No scan results")
+		}
 		currentScanResult = scanHistory.ScanResults[0]
-		if err := g.SetCurrentView("side"); err != nil {
+		if _, err := g.SetCurrentView("side"); err != nil {
 			return err
 		}
 	}
@@ -533,7 +530,7 @@ func setSummaryLayout(g *gocui.Gui) error {
 			return err
 		}
 
-		lines := summaryLines(currentScanResult)
+		lines := summaryLines()
 		fmt.Fprintf(v, lines)
 
 		v.Highlight = true
@@ -543,21 +540,21 @@ func setSummaryLayout(g *gocui.Gui) error {
 	return nil
 }
 
-func summaryLines(data models.ScanResult) string {
+func summaryLines() string {
 	stable := uitable.New()
 	stable.MaxColWidth = 1000
 	stable.Wrap = false
 
 	indexFormat := ""
-	if len(data.KnownCves) < 10 {
+	if len(currentScanResult.AllCves()) < 10 {
 		indexFormat = "[%1d]"
-	} else if len(data.KnownCves) < 100 {
+	} else if len(currentScanResult.AllCves()) < 100 {
 		indexFormat = "[%2d]"
 	} else {
 		indexFormat = "[%3d]"
 	}
 
-	for i, d := range data.KnownCves {
+	for i, d := range currentScanResult.AllCves() {
 		var cols []string
 		//  packs := []string{}
 		//  for _, pack := range d.Packages {
@@ -643,20 +640,21 @@ type dataForTmpl struct {
 	CvssVector       string
 	CvssSeverity     string
 	Summary          string
+	CweURL           string
 	VulnSiteLinks    []string
 	References       []cve.Reference
 	Packages         []string
-	CpeNames         []models.CpeName
+	CpeNames         []string
 	PublishedDate    time.Time
 	LastModifiedDate time.Time
 }
 
 func detailLines() (string, error) {
-	if len(currentScanResult.KnownCves) == 0 {
+	if len(currentScanResult.AllCves()) == 0 {
 		return "No vulnerable packages", nil
 	}
 
-	cveInfo := currentScanResult.KnownCves[currentCveInfo]
+	cveInfo := currentScanResult.AllCves()[currentCveInfo]
 	cveID := cveInfo.CveDetail.CveID
 
 	tmpl, err := template.New("detail").Parse(detailTemplate())
@@ -682,6 +680,8 @@ func detailLines() (string, error) {
 		refs = nvd.VulnSiteReferences()
 	}
 
+	cweURL := cweURL(cveInfo.CveDetail.CweID())
+
 	links := []string{
 		fmt.Sprintf("[NVD]( %s )", fmt.Sprintf("%s?vulnId=%s", nvdBaseURL, cveID)),
 		fmt.Sprintf("[MITRE]( %s )", fmt.Sprintf("%s%s", mitreBaseURL, cveID)),
@@ -715,6 +715,7 @@ func detailLines() (string, error) {
 		CvssSeverity:  cvssSeverity,
 		CvssVector:    cvssVector,
 		Summary:       summary,
+		CweURL:        cweURL,
 		VulnSiteLinks: links,
 		References:    refs,
 		Packages:      packages,
@@ -746,14 +747,19 @@ Summary
 
  {{.Summary }}
 
+CWE
+--------------
+
+ {{.CweURL }}
+
 Package/CPE
 --------------
 
 {{range $pack := .Packages -}}
 * {{$pack}}
 {{end -}}
-{{range .CpeNames -}}
-* {{.Name}}
+{{range $name := .CpeNames -}}
+* {{$name}}
 {{end}}
 Links
 --------------

report/util.go

@@ -20,88 +20,49 @@ package report
 import (
 	"bytes"
 	"fmt"
-	"os"
-	"path/filepath"
 	"strings"
-	"time"
 
 	"github.com/future-architect/vuls/config"
 	"github.com/future-architect/vuls/models"
 	"github.com/gosuri/uitable"
 )
 
-func ensureResultDir() (path string, err error) {
-	if resultDirPath != "" {
-		return resultDirPath, nil
-	}
+const maxColWidth = 80
 
-	const timeLayout = "20060102_1504"
-	timedir := time.Now().Format(timeLayout)
-	wd, _ := os.Getwd()
-	dir := filepath.Join(wd, "results", timedir)
-	if err := os.MkdirAll(dir, 0755); err != nil {
-		return "", fmt.Errorf("Failed to create dir: %s", err)
-	}
-
-	symlinkPath := filepath.Join(wd, "results", "current")
-	if _, err := os.Stat(symlinkPath); err == nil {
-		if err := os.Remove(symlinkPath); err != nil {
-			return "", fmt.Errorf(
-				"Failed to remove symlink. path: %s, err: %s", symlinkPath, err)
+func toScanSummary(rs ...models.ScanResult) string {
+	table := uitable.New()
+	table.MaxColWidth = maxColWidth
+	table.Wrap = true
+	for _, r := range rs {
+		cols := []interface{}{
+			r.ServerName,
+			fmt.Sprintf("%s%s", r.Family, r.Release),
+			fmt.Sprintf("%d CVEs", len(r.ScannedCves)),
+			r.Packages.ToUpdatablePacksSummary(),
 		}
+		table.AddRow(cols...)
 	}
-
-	if err := os.Symlink(dir, symlinkPath); err != nil {
-		return "", fmt.Errorf(
-			"Failed to create symlink: path: %s, err: %s", symlinkPath, err)
-	}
-	return dir, nil
+	return fmt.Sprintf("%s\n", table)
 }
 
-func toPlainText(scanResult models.ScanResult) (string, error) {
-	serverInfo := scanResult.ServerInfo()
-
-	var buffer bytes.Buffer
-	for i := 0; i < len(serverInfo); i++ {
-		buffer.WriteString("=")
+func toOneLineSummary(rs ...models.ScanResult) string {
+	table := uitable.New()
+	table.MaxColWidth = maxColWidth
+	table.Wrap = true
+	for _, r := range rs {
+		cols := []interface{}{
+			r.ServerName,
+			r.CveSummary(),
+			r.Packages.ToUpdatablePacksSummary(),
+		}
+		table.AddRow(cols...)
 	}
-	header := fmt.Sprintf("%s\n%s", serverInfo, buffer.String())
-
-	if len(scanResult.KnownCves) == 0 && len(scanResult.UnknownCves) == 0 {
-		return fmt.Sprintf(`
-%s
-No unsecure packages.
-`, header), nil
-	}
-
-	summary := ToPlainTextSummary(scanResult)
-	scoredReport, unscoredReport := []string{}, []string{}
-	scoredReport, unscoredReport = toPlainTextDetails(scanResult, scanResult.Family)
-
-	scored := strings.Join(scoredReport, "\n\n")
-
-	unscored := ""
-	if !config.Conf.IgnoreUnscoredCves {
-		unscored = strings.Join(unscoredReport, "\n\n")
-	}
-
-	detail := fmt.Sprintf(`
-%s
-
-%s
-`,
-		scored,
-		unscored,
-	)
-	text := fmt.Sprintf("%s\n%s\n%s\n", header, summary, detail)
-
-	return text, nil
+	return fmt.Sprintf("%s\n", table)
 }
 
-// ToPlainTextSummary format summary for plain text.
-func ToPlainTextSummary(r models.ScanResult) string {
+func toShortPlainText(r models.ScanResult) string {
 	stable := uitable.New()
-	stable.MaxColWidth = 84
+	stable.MaxColWidth = maxColWidth
 	stable.Wrap = true
 
 	cves := r.KnownCves
@@ -109,14 +70,45 @@ func ToPlainTextSummary(r models.ScanResult) string {
 		cves = append(cves, r.UnknownCves...)
 	}
 
-	for _, d := range cves {
-		var scols []string
+	var buf bytes.Buffer
+	for i := 0; i < len(r.ServerInfo()); i++ {
+		buf.WriteString("=")
+	}
+	header := fmt.Sprintf("%s\n%s\n%s\t%s\n\n",
+		r.ServerInfo(),
+		buf.String(),
+		r.CveSummary(),
+		r.Packages.ToUpdatablePacksSummary(),
+	)
 
+	if len(cves) == 0 {
+		return fmt.Sprintf(`
+%s
+No CVE-IDs are found in updatable packages.
+%s
+`, header, r.Packages.ToUpdatablePacksSummary())
+	}
+
+	for _, d := range cves {
+		var packsVer string
+		for _, p := range d.Packages {
+			packsVer += fmt.Sprintf(
+				"%s -> %s\n", p.ToStringCurrentVersion(), p.ToStringNewVersion())
+		}
+		for _, n := range d.CpeNames {
+			packsVer += n
+		}
+
+		var scols []string
 		switch {
 		case config.Conf.Lang == "ja" &&
 			0 < d.CveDetail.Jvn.CvssScore():
-
-			summary := d.CveDetail.Jvn.CveTitle()
+			summary := fmt.Sprintf("%s\n%s\n%s\n%s",
+				d.CveDetail.Jvn.CveTitle(),
+				d.CveDetail.Jvn.Link(),
+				distroLinks(d, r.Family)[0].url,
+				packsVer,
+			)
 			scols = []string{
 				d.CveDetail.CveID,
 				fmt.Sprintf("%-4.1f (%s)",
@@ -125,8 +117,15 @@ func ToPlainTextSummary(r models.ScanResult) string {
 				),
 				summary,
 			}
+
 		case 0 < d.CveDetail.CvssScore("en"):
-			summary := d.CveDetail.Nvd.CveSummary()
+			summary := fmt.Sprintf("%s\n%s/%s\n%s\n%s",
+				d.CveDetail.Nvd.CveSummary(),
+				cveDetailsBaseURL,
+				d.CveDetail.CveID,
+				distroLinks(d, r.Family)[0].url,
+				packsVer,
+			)
 			scols = []string{
 				d.CveDetail.CveID,
 				fmt.Sprintf("%-4.1f (%s)",
@@ -136,10 +135,12 @@ func ToPlainTextSummary(r models.ScanResult) string {
 				summary,
 			}
 		default:
+			summary := fmt.Sprintf("%s\n%s",
+				distroLinks(d, r.Family)[0].url, packsVer)
 			scols = []string{
 				d.CveDetail.CveID,
 				"?",
-				d.CveDetail.Nvd.CveSummary(),
+				summary,
 			}
 		}
 
@@ -148,12 +149,55 @@ func ToPlainTextSummary(r models.ScanResult) string {
 			cols[i] = scols[i]
 		}
 		stable.AddRow(cols...)
+		stable.AddRow("")
 	}
-	return fmt.Sprintf("%s", stable)
+	return fmt.Sprintf("%s\n%s\n", header, stable)
 }
 
-func toPlainTextDetails(data models.ScanResult, osFamily string) (scoredReport, unscoredReport []string) {
-	for _, cve := range data.KnownCves {
+func toFullPlainText(r models.ScanResult) string {
+	serverInfo := r.ServerInfo()
+
+	var buf bytes.Buffer
+	for i := 0; i < len(serverInfo); i++ {
+		buf.WriteString("=")
+	}
+	header := fmt.Sprintf("%s\n%s\n%s\t%s\n",
+		r.ServerInfo(),
+		buf.String(),
+		r.CveSummary(),
+		r.Packages.ToUpdatablePacksSummary(),
+	)
+
+	if len(r.KnownCves) == 0 && len(r.UnknownCves) == 0 {
+		return fmt.Sprintf(`
+%s
+No CVE-IDs are found in updatable packages.
+%s
+`, header, r.Packages.ToUpdatablePacksSummary())
+	}
+
+	scoredReport, unscoredReport := []string{}, []string{}
+	scoredReport, unscoredReport = toPlainTextDetails(r, r.Family)
+
+	unscored := ""
+	if !config.Conf.IgnoreUnscoredCves {
+		unscored = strings.Join(unscoredReport, "\n\n")
+	}
+
+	scored := strings.Join(scoredReport, "\n\n")
+	detail := fmt.Sprintf(`
+%s
+
+%s
+`,
+		scored,
+		unscored,
+	)
+	return fmt.Sprintf("%s\n%s\n", header, detail)
+}
+
+func toPlainTextDetails(r models.ScanResult, osFamily string) (scoredReport, unscoredReport []string) {
+	for _, cve := range r.KnownCves {
 		switch config.Conf.Lang {
 		case "en":
 			if 0 < cve.CveDetail.Nvd.CvssScore() {
@@ -176,7 +220,7 @@ func toPlainTextDetails(data models.ScanResult, osFamily string) (scoredReport,
 			}
 		}
 	}
-	for _, cve := range data.UnknownCves {
+	for _, cve := range r.UnknownCves {
 		unscoredReport = append(
 			unscoredReport, toPlainTextUnknownCve(cve, osFamily))
 	}
@@ -186,7 +230,7 @@ func toPlainTextDetails(data models.ScanResult, osFamily string) (scoredReport,
 func toPlainTextUnknownCve(cveInfo models.CveInfo, osFamily string) string {
 	cveID := cveInfo.CveDetail.CveID
 	dtable := uitable.New()
-	dtable.MaxColWidth = 100
+	dtable.MaxColWidth = maxColWidth
 	dtable.Wrap = true
 	dtable.AddRow(cveID)
 	dtable.AddRow("-------------")
@@ -200,19 +244,19 @@ func toPlainTextUnknownCve(cveInfo models.CveInfo, osFamily string) string {
 	for _, link := range dlinks {
 		dtable.AddRow(link.title, link.url)
 	}
+	dtable = addPackageInfos(dtable, cveInfo.Packages)
+	dtable = addCpeNames(dtable, cveInfo.CpeNames)
 
 	return fmt.Sprintf("%s", dtable)
 }
 
 func toPlainTextDetailsLangJa(cveInfo models.CveInfo, osFamily string) string {
-
 	cveDetail := cveInfo.CveDetail
 	cveID := cveDetail.CveID
 	jvn := cveDetail.Jvn
 
 	dtable := uitable.New()
-	//TODO resize
-	dtable.MaxColWidth = 100
+	dtable.MaxColWidth = maxColWidth
 	dtable.Wrap = true
 	dtable.AddRow(cveID)
 	dtable.AddRow("-------------")
@@ -228,6 +272,8 @@ func toPlainTextDetailsLangJa(cveInfo models.CveInfo, osFamily string) string {
 	dtable.AddRow("Vector", jvn.CvssVector())
 	dtable.AddRow("Title", jvn.CveTitle())
 	dtable.AddRow("Description", jvn.CveSummary())
+	dtable.AddRow(cveDetail.CweID(), cweURL(cveDetail.CweID()))
+	dtable.AddRow(cveDetail.CweID()+"(JVN)", cweJvnURL(cveDetail.CweID()))
 
 	dtable.AddRow("JVN", jvn.Link())
 	dtable.AddRow("NVD", fmt.Sprintf("%s?vulnId=%s", nvdBaseURL, cveID))
@@ -252,8 +298,7 @@ func toPlainTextDetailsLangEn(d models.CveInfo, osFamily string) string {
 	nvd := cveDetail.Nvd
 
 	dtable := uitable.New()
-	//TODO resize
-	dtable.MaxColWidth = 100
+	dtable.MaxColWidth = maxColWidth
 	dtable.Wrap = true
 	dtable.AddRow(cveID)
 	dtable.AddRow("-------------")
@@ -270,6 +315,8 @@ func toPlainTextDetailsLangEn(d models.CveInfo, osFamily string) string {
 
 	dtable.AddRow("Vector", nvd.CvssVector())
 	dtable.AddRow("Summary", nvd.CveSummary())
+	dtable.AddRow("CWE", cweURL(cveDetail.CweID()))
+
 	dtable.AddRow("NVD", fmt.Sprintf("%s?vulnId=%s", nvdBaseURL, cveID))
 	dtable.AddRow("MITRE", fmt.Sprintf("%s%s", mitreBaseURL, cveID))
 	dtable.AddRow("CVE Details", fmt.Sprintf("%s/%s", cveDetailsBaseURL, cveID))
@@ -355,13 +402,12 @@ func distroLinks(cveInfo models.CveInfo, osFamily string) []distroLink {
 	}
 }
 
-//TODO
 // addPackageInfos add package information related the CVE to table
 func addPackageInfos(table *uitable.Table, packs []models.PackageInfo) *uitable.Table {
 	for i, p := range packs {
 		var title string
 		if i == 0 {
-			title = "Package/CPE"
+			title = "Package"
 		}
 		ver := fmt.Sprintf(
 			"%s -> %s", p.ToStringCurrentVersion(), p.ToStringNewVersion())
@@ -370,9 +416,18 @@ func addPackageInfos(table *uitable.Table, packs []models.PackageInfo) *uitable.
 	return table
 }
 
-func addCpeNames(table *uitable.Table, names []models.CpeName) *uitable.Table {
-	for _, p := range names {
-		table.AddRow("CPE", fmt.Sprintf("%s", p.Name))
+func addCpeNames(table *uitable.Table, names []string) *uitable.Table {
+	for _, n := range names {
+		table.AddRow("CPE", fmt.Sprintf("%s", n))
 	}
 	return table
 }
+
+func cweURL(cweID string) string {
+	return fmt.Sprintf("https://cwe.mitre.org/data/definitions/%s.html",
+		strings.TrimPrefix(cweID, "CWE-"))
+}
+
+func cweJvnURL(cweID string) string {
+	return fmt.Sprintf("http://jvndb.jvn.jp/ja/cwe/%s.html", cweID)
+}

report/writer.go

@@ -17,7 +17,12 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 package report
 
-import "github.com/future-architect/vuls/models"
+import (
+	"bytes"
+	"compress/gzip"
+
+	"github.com/future-architect/vuls/models"
+)
 
 const (
 	nvdBaseURL            = "https://web.nvd.nist.gov/view/vuln/detail"
@@ -33,11 +38,27 @@ const (
 	debianTrackerBaseURL  = "https://security-tracker.debian.org/tracker"
 
 	freeBSDVuXMLBaseURL = "https://vuxml.freebsd.org/freebsd/%s.html"
+
+	vulsOpenTag  = "<vulsreport>"
+	vulsCloseTag = "</vulsreport>"
 )
 
 // ResultWriter Interface
 type ResultWriter interface {
-	Write([]models.ScanResult) error
+	Write(...models.ScanResult) error
 }
 
-var resultDirPath string
+func gz(data []byte) ([]byte, error) {
+	var b bytes.Buffer
+	gz := gzip.NewWriter(&b)
+	if _, err := gz.Write(data); err != nil {
+		return nil, err
+	}
+	if err := gz.Flush(); err != nil {
+		return nil, err
+	}
+	if err := gz.Close(); err != nil {
+		return nil, err
+	}
+	return b.Bytes(), nil
+}

scan/base.go

@@ -19,21 +19,22 @@ package scan
 
 import (
 	"fmt"
+	"regexp"
 	"sort"
 	"strings"
+	"time"
 
 	"github.com/Sirupsen/logrus"
 	"github.com/future-architect/vuls/config"
-	"github.com/future-architect/vuls/cveapi"
 	"github.com/future-architect/vuls/models"
 )
 
 type base struct {
 	ServerInfo config.ServerInfo
+	Distro     config.Distro
+	Platform   models.Platform
 
-	Family   string
-	Release  string
-	Platform models.Platform
+	lackDependencies []string
 	osPackages
 
 	log  *logrus.Entry
@@ -52,13 +53,20 @@ func (l base) getServerInfo() config.ServerInfo {
 	return l.ServerInfo
 }
 
-func (l *base) setDistributionInfo(fam, rel string) {
-	l.Family = fam
-	l.Release = rel
+func (l *base) setDistro(fam, rel string) {
+	d := config.Distro{
+		Family:  fam,
+		Release: rel,
+	}
+	l.Distro = d
+
+	s := l.getServerInfo()
+	s.Distro = d
+	l.setServerInfo(s)
 }
 
-func (l base) getDistributionInfo() string {
-	return fmt.Sprintf("%s %s", l.Family, l.Release)
+func (l base) getDistro() config.Distro {
+	return l.Distro
 }
 
 func (l *base) setPlatform(p models.Platform) {
@@ -69,6 +77,10 @@ func (l base) getPlatform() models.Platform {
 	return l.Platform
 }
 
+func (l base) getLackDependencies() []string {
+	return l.lackDependencies
+}
+
 func (l base) allContainers() (containers []config.Container, err error) {
 	switch l.ServerInfo.Container.Type {
 	case "", "docker":
@@ -164,9 +176,7 @@ func (l base) detectRunningOnAws() (ok bool, instanceID string, err error) {
 		r := l.ssh(cmd, noSudo)
 		if r.isSuccess() {
 			id := strings.TrimSpace(r.Stdout)
-
-			if id == "not found" {
-				// status: 0, stdout: "not found" on degitalocean or Azure
+			if !l.isAwsInstanceID(id) {
 				return false, "", nil
 			}
 			return true, id, nil
@@ -186,6 +196,9 @@ func (l base) detectRunningOnAws() (ok bool, instanceID string, err error) {
 		r := l.ssh(cmd, noSudo)
 		if r.isSuccess() {
 			id := strings.TrimSpace(r.Stdout)
+			if !l.isAwsInstanceID(id) {
+				return false, "", nil
+			}
 			return true, id, nil
 		}
 
@@ -202,92 +215,37 @@ func (l base) detectRunningOnAws() (ok bool, instanceID string, err error) {
 		l.ServerInfo.ServerName, l.ServerInfo.Container.Name)
 }
 
+// http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/resource-ids.html
+var awsInstanceIDPattern = regexp.MustCompile(`^i-[0-9a-f]+$`)
+
+func (l base) isAwsInstanceID(str string) bool {
+	return awsInstanceIDPattern.MatchString(str)
+}
+
 func (l *base) convertToModel() (models.ScanResult, error) {
-	var scoredCves, unscoredCves models.CveInfos
-	for _, p := range l.UnsecurePackages {
-		if p.CveDetail.CvssScore(config.Conf.Lang) <= 0 {
-			unscoredCves = append(unscoredCves, models.CveInfo{
-				CveDetail:        p.CveDetail,
-				Packages:         p.Packs,
-				DistroAdvisories: p.DistroAdvisories, // only Amazon Linux
-			})
-			continue
-		}
-
-		cpenames := []models.CpeName{}
-		for _, cpename := range p.CpeNames {
-			cpenames = append(cpenames,
-				models.CpeName{Name: cpename})
-		}
-
-		cve := models.CveInfo{
-			CveDetail:        p.CveDetail,
-			Packages:         p.Packs,
-			DistroAdvisories: p.DistroAdvisories, // only Amazon Linux
-			CpeNames:         cpenames,
-		}
-		scoredCves = append(scoredCves, cve)
+	for _, p := range l.VulnInfos {
+		sort.Sort(models.PackageInfosByName(p.Packages))
 	}
+	sort.Sort(l.VulnInfos)
 
 	container := models.Container{
 		ContainerID: l.ServerInfo.Container.ContainerID,
 		Name:        l.ServerInfo.Container.Name,
 	}
 
-	sort.Sort(scoredCves)
-	sort.Sort(unscoredCves)
-
 	return models.ScanResult{
 		ServerName:  l.ServerInfo.ServerName,
-		Family:      l.Family,
-		Release:     l.Release,
+		ScannedAt:   time.Now(),
+		Family:      l.Distro.Family,
+		Release:     l.Distro.Release,
 		Container:   container,
 		Platform:    l.Platform,
-		KnownCves:   scoredCves,
-		UnknownCves: unscoredCves,
+		ScannedCves: l.VulnInfos,
+		Packages:    l.Packages,
 		Optional:    l.ServerInfo.Optional,
 	}, nil
 }
 
-// scanVulnByCpeName search vulnerabilities that specified in config file.
-func (l *base) scanVulnByCpeName() error {
-	unsecurePacks := CvePacksList{}
-
-	serverInfo := l.getServerInfo()
-	cpeNames := serverInfo.CpeNames
-
-	// remove duplicate
-	set := map[string]CvePacksInfo{}
-
-	for _, name := range cpeNames {
-		details, err := cveapi.CveClient.FetchCveDetailsByCpeName(name)
-		if err != nil {
-			return err
-		}
-		for _, detail := range details {
-			if val, ok := set[detail.CveID]; ok {
-				names := val.CpeNames
-				names = append(names, name)
-				val.CpeNames = names
-				set[detail.CveID] = val
-			} else {
-				set[detail.CveID] = CvePacksInfo{
-					CveID:     detail.CveID,
-					CveDetail: detail,
-					CpeNames:  []string{name},
-				}
-			}
-		}
-	}
-
-	for key := range set {
-		unsecurePacks = append(unsecurePacks, set[key])
-	}
-	unsecurePacks = append(unsecurePacks, l.UnsecurePackages...)
-	l.setUnsecurePackages(unsecurePacks)
-	return nil
-}
-
 func (l *base) setErrs(errs []error) {
 	l.errs = errs
 }

scan/base_test.go

@@ -56,3 +56,25 @@ f570ae647edc agitated_lovelace`,
 		}
 	}
 }
+
+func TestIsAwsInstanceID(t *testing.T) {
+	var tests = []struct {
+		in       string
+		expected bool
+	}{
+		{"i-1234567a", true},
+		{"i-1234567890abcdef0", true},
+		{"i-1234567890abcdef0000000", true},
+		{"e-1234567890abcdef0", false},
+		{"i-1234567890abcdef0 foo bar", false},
+		{"no data", false},
+	}
+
+	r := newRedhat(config.ServerInfo{})
+	for _, tt := range tests {
+		actual := r.isAwsInstanceID(tt.in)
+		if tt.expected != actual {
+			t.Errorf("expected %t, actual %t, str: %s", tt.expected, actual, tt.in)
+		}
+	}
+}

scan/debian.go

@@ -24,8 +24,8 @@ import (
 	"strings"
 	"time"
 
+	"github.com/future-architect/vuls/cache"
 	"github.com/future-architect/vuls/config"
-	"github.com/future-architect/vuls/cveapi"
 	"github.com/future-architect/vuls/models"
 	"github.com/future-architect/vuls/util"
 )
@@ -39,6 +39,7 @@ type debian struct {
 func newDebian(c config.ServerInfo) *debian {
 	d := &debian{}
 	d.log = util.NewCustomLogger(c)
+	d.setServerInfo(c)
 	return d
 }
 
@@ -46,7 +47,6 @@ func newDebian(c config.ServerInfo) *debian {
 // https://github.com/serverspec/specinfra/blob/master/lib/specinfra/helper/detect_os/debian.rb
 func detectDebian(c config.ServerInfo) (itsMe bool, deb osTypeInterface, err error) {
 	deb = newDebian(c)
-	deb.setServerInfo(c)
 
 	if r := sshExec(c, "ls /etc/debian_version", noSudo); !r.isSuccess() {
 		if r.Error != nil {
@@ -65,17 +65,16 @@ func detectDebian(c config.ServerInfo) (itsMe bool, deb osTypeInterface, err err
 		//  root@fa3ec524be43:/# lsb_release -ir
 		//  Distributor ID:	Ubuntu
 		//  Release:	14.04
-		re, _ := regexp.Compile(
-			`(?s)^Distributor ID:\s*(.+?)\n*Release:\s*(.+?)$`)
+		re := regexp.MustCompile(`(?s)^Distributor ID:\s*(.+?)\n*Release:\s*(.+?)$`)
 		result := re.FindStringSubmatch(trim(r.Stdout))
 
 		if len(result) == 0 {
-			deb.setDistributionInfo("debian/ubuntu", "unknown")
+			deb.setDistro("debian/ubuntu", "unknown")
 			Log.Warnf(
 				"Unknown Debian/Ubuntu version. lsb_release -ir: %s", r)
 		} else {
 			distro := strings.ToLower(trim(result[1]))
-			deb.setDistributionInfo(distro, trim(result[2]))
+			deb.setDistro(distro, trim(result[2]))
 		}
 		return true, deb, nil
 	}
@@ -86,16 +85,15 @@ func detectDebian(c config.ServerInfo) (itsMe bool, deb osTypeInterface, err err
 		//  DISTRIB_RELEASE=14.04
 		//  DISTRIB_CODENAME=trusty
 		//  DISTRIB_DESCRIPTION="Ubuntu 14.04.2 LTS"
-		re, _ := regexp.Compile(
-			`(?s)^DISTRIB_ID=(.+?)\n*DISTRIB_RELEASE=(.+?)\n.*$`)
+		re := regexp.MustCompile(`(?s)^DISTRIB_ID=(.+?)\n*DISTRIB_RELEASE=(.+?)\n.*$`)
 		result := re.FindStringSubmatch(trim(r.Stdout))
 		if len(result) == 0 {
 			Log.Warnf(
 				"Unknown Debian/Ubuntu. cat /etc/lsb-release: %s", r)
-			deb.setDistributionInfo("debian/ubuntu", "unknown")
+			deb.setDistro("debian/ubuntu", "unknown")
 		} else {
 			distro := strings.ToLower(trim(result[1]))
-			deb.setDistributionInfo(distro, trim(result[2]))
+			deb.setDistro(distro, trim(result[2]))
 		}
 		return true, deb, nil
 	}
@@ -103,7 +101,7 @@ func detectDebian(c config.ServerInfo) (itsMe bool, deb osTypeInterface, err err
 	// Debian
 	cmd := "cat /etc/debian_version"
 	if r := sshExec(c, cmd, noSudo); r.isSuccess() {
-		deb.setDistributionInfo("debian", trim(r.Stdout))
+		deb.setDistro("debian", trim(r.Stdout))
 		return true, deb, nil
 	}
 
@@ -125,7 +123,30 @@ func (o *debian) checkIfSudoNoPasswd() error {
 	return nil
 }
 
+func (o *debian) checkDependencies() error {
+	switch o.Distro.Family {
+	case "ubuntu":
+		return nil
+
+	case "debian":
+		// Debian needs aptitude to get changelogs.
+		// Because unable to get changelogs via apt-get changelog on Debian.
+		name := "aptitude"
+		cmd := name + " -h"
+		if r := o.ssh(cmd, noSudo); !r.isSuccess() {
+			o.lackDependencies = []string{name}
+		}
+		return nil
+
+	default:
+		return fmt.Errorf("Not implemented yet: %s", o.Distro)
+	}
+}
+
 func (o *debian) install() error {
+	if len(o.lackDependencies) == 0 {
+		return nil
+	}
 
 	// apt-get update
 	o.log.Infof("apt-get update...")
@@ -136,37 +157,15 @@ func (o *debian) install() error {
 		return fmt.Errorf(msg)
 	}
 
-	if o.Family == "debian" {
-		// install aptitude
-		cmd = util.PrependProxyEnv("apt-get install --force-yes -y aptitude")
+	for _, name := range o.lackDependencies {
+		cmd = util.PrependProxyEnv("apt-get install -y " + name)
 		if r := o.ssh(cmd, sudo); !r.isSuccess() {
 			msg := fmt.Sprintf("Failed to SSH: %s", r)
 			o.log.Errorf(msg)
 			return fmt.Errorf(msg)
 		}
-		o.log.Infof("Installed: aptitude")
+		o.log.Infof("Installed: " + name)
 	}
-
-	// install unattended-upgrades
-	if !config.Conf.UseUnattendedUpgrades {
-		return nil
-	}
-
-	if r := o.ssh("type unattended-upgrade", noSudo); r.isSuccess() {
-		o.log.Infof(
-			"Ignored: unattended-upgrade already installed")
-		return nil
-	}
-
-	cmd = util.PrependProxyEnv(
-		"apt-get install --force-yes -y unattended-upgrades")
-	if r := o.ssh(cmd, sudo); !r.isSuccess() {
-		msg := fmt.Sprintf("Failed to SSH: %s", r)
-		o.log.Errorf(msg)
-		return fmt.Errorf(msg)
-	}
-
-	o.log.Infof("Installed: unattended-upgrades")
 	return nil
 }
 
@@ -179,12 +178,12 @@ func (o *debian) scanPackages() error {
 	}
 	o.setPackages(packs)
 
-	var unsecurePacks []CvePacksInfo
+	var unsecurePacks []models.VulnInfo
 	if unsecurePacks, err = o.scanUnsecurePackages(packs); err != nil {
 		o.log.Errorf("Failed to scan vulnerable packages")
 		return err
 	}
-	o.setUnsecurePackages(unsecurePacks)
+	o.setVulnInfos(unsecurePacks)
 	return nil
 }
 
@@ -214,12 +213,16 @@ func (o *debian) scanInstalledPackages() (packs []models.PackageInfo, err error)
 	return
 }
 
+var packageLinePattern = regexp.MustCompile(`^([^\t']+)\t(.+)$`)
+
 func (o *debian) parseScannedPackagesLine(line string) (name, version string, err error) {
-	re, _ := regexp.Compile(`^([^\t']+)\t(.+)$`)
-	result := re.FindStringSubmatch(line)
+	result := packageLinePattern.FindStringSubmatch(line)
 	if len(result) == 3 {
 		// remove :amd64, i386...
-		name = regexp.MustCompile(":.+").ReplaceAllString(result[1], "")
+		name = result[1]
+		if i := strings.IndexRune(name, ':'); i >= 0 {
+			name = name[:i]
+		}
 		version = result[2]
 		return
 	}
@@ -227,173 +230,126 @@ func (o *debian) parseScannedPackagesLine(line string) (name, version string, er
 	return "", "", fmt.Errorf("Unknown format: %s", line)
 }
 
-//  unattended-upgrade command need to check security upgrades).
 func (o *debian) checkRequiredPackagesInstalled() error {
-
-	if o.Family == "debian" {
+	if o.Distro.Family == "debian" {
 		if r := o.ssh("test -f /usr/bin/aptitude", noSudo); !r.isSuccess() {
 			msg := fmt.Sprintf("aptitude is not installed: %s", r)
 			o.log.Errorf(msg)
 			return fmt.Errorf(msg)
 		}
 	}
-
-	if !config.Conf.UseUnattendedUpgrades {
-		return nil
-	}
-
-	if r := o.ssh("type unattended-upgrade", noSudo); !r.isSuccess() {
-		msg := fmt.Sprintf("unattended-upgrade is not installed: %s", r)
-		o.log.Errorf(msg)
-		return fmt.Errorf(msg)
-	}
 	return nil
 }
 
-//TODO return whether already expired.
-func (o *debian) scanUnsecurePackages(packs []models.PackageInfo) ([]CvePacksInfo, error) {
-	//  cmd := prependProxyEnv(conf.HTTPProxy, "apt-get update | cat; echo 1")
+func (o *debian) scanUnsecurePackages(installed []models.PackageInfo) ([]models.VulnInfo, error) {
+	o.log.Infof("apt-get update...")
 	cmd := util.PrependProxyEnv("apt-get update")
 	if r := o.ssh(cmd, sudo); !r.isSuccess() {
 		return nil, fmt.Errorf("Failed to SSH: %s", r)
 	}
 
-	var upgradablePackNames []string
-	var err error
-	if config.Conf.UseUnattendedUpgrades {
-		upgradablePackNames, err = o.GetUnsecurePackNamesUsingUnattendedUpgrades()
-		if err != nil {
-			return []CvePacksInfo{}, err
-		}
-	} else {
-		upgradablePackNames, err = o.GetUpgradablePackNames()
-		if err != nil {
-			return []CvePacksInfo{}, err
-		}
+	// Convert the name of upgradable packages to PackageInfo struct
+	upgradableNames, err := o.GetUpgradablePackNames()
+	if err != nil {
+		return nil, err
 	}
-
-	// Convert package name to PackageInfo struct
-	var unsecurePacks []models.PackageInfo
-	for _, name := range upgradablePackNames {
-		for _, pack := range packs {
+	var upgradablePacks []models.PackageInfo
+	for _, name := range upgradableNames {
+		for _, pack := range installed {
 			if pack.Name == name {
-				unsecurePacks = append(unsecurePacks, pack)
+				upgradablePacks = append(upgradablePacks, pack)
 				break
 			}
 		}
 	}
 
-	unsecurePacks, err = o.fillCandidateVersion(unsecurePacks)
+	// Fill the candidate versions of upgradable packages
+	upgradablePacks, err = o.fillCandidateVersion(upgradablePacks)
 	if err != nil {
 		return nil, fmt.Errorf("Failed to fill candidate versions. err: %s", err)
 	}
 
+	o.Packages.MergeNewVersion(upgradablePacks)
+
+	// Setup changelog cache
+	current := cache.Meta{
+		Name:   o.getServerInfo().GetServerName(),
+		Distro: o.getServerInfo().Distro,
+		Packs:  upgradablePacks,
+	}
+	o.log.Debugf("Ensure changelog cache: %s", current.Name)
+	if err := o.ensureChangelogCache(current); err != nil {
+		return nil, err
+	}
+
 	// Collect CVE information of upgradable packages
-	cvePacksInfos, err := o.scanPackageCveInfos(unsecurePacks)
+	vulnInfos, err := o.scanVulnInfos(upgradablePacks)
 	if err != nil {
 		return nil, fmt.Errorf("Failed to scan unsecure packages. err: %s", err)
 	}
 
-	return cvePacksInfos, nil
+	return vulnInfos, nil
 }
 
-func (o *debian) fillCandidateVersion(packs []models.PackageInfo) ([]models.PackageInfo, error) {
-	reqChan := make(chan models.PackageInfo, len(packs))
-	resChan := make(chan models.PackageInfo, len(packs))
-	errChan := make(chan error, len(packs))
-	defer close(resChan)
-	defer close(errChan)
-	defer close(reqChan)
-
-	go func() {
-		for _, pack := range packs {
-			reqChan <- pack
-		}
-	}()
-
-	timeout := time.After(5 * 60 * time.Second)
-	concurrency := 5
-	tasks := util.GenWorkers(concurrency)
-	for range packs {
-		tasks <- func() {
-			select {
-			case pack := <-reqChan:
-				func(p models.PackageInfo) {
-					cmd := fmt.Sprintf("LANG=en_US.UTF-8 apt-cache policy %s", p.Name)
-					r := o.ssh(cmd, sudo)
-					if !r.isSuccess() {
-						errChan <- fmt.Errorf("Failed to SSH: %s.", r)
-						return
-					}
-					ver, err := o.parseAptCachePolicy(r.Stdout, p.Name)
-					if err != nil {
-						errChan <- fmt.Errorf("Failed to parse %s", err)
-					}
-					p.NewVersion = ver.Candidate
-					resChan <- p
-				}(pack)
-			}
-		}
-	}
-
-	errs := []error{}
-	result := []models.PackageInfo{}
-	for i := 0; i < len(packs); i++ {
-		select {
-		case pack := <-resChan:
-			result = append(result, pack)
-			o.log.Infof("(%d/%d) Upgradable: %s-%s -> %s",
-				i+1, len(packs), pack.Name, pack.Version, pack.NewVersion)
-		case err := <-errChan:
-			errs = append(errs, err)
-		case <-timeout:
-			return nil, fmt.Errorf("Timeout fillCandidateVersion")
-		}
-	}
-	if 0 < len(errs) {
-		return nil, fmt.Errorf("%v", errs)
-	}
-	return result, nil
-}
-
-func (o *debian) GetUnsecurePackNamesUsingUnattendedUpgrades() (packNames []string, err error) {
-	cmd := util.PrependProxyEnv("unattended-upgrades --dry-run -d 2>&1 ")
-	release, err := strconv.ParseFloat(o.Release, 64)
+func (o *debian) ensureChangelogCache(current cache.Meta) error {
+	// Search from cache
+	old, found, err := cache.DB.GetMeta(current.Name)
 	if err != nil {
-		return packNames, fmt.Errorf(
-			"OS Release Version is invalid, %s, %s", o.Family, o.Release)
+		return fmt.Errorf("Failed to get meta. err: %s", err)
 	}
-	switch {
-	case release < 12:
-		return packNames, fmt.Errorf(
-			"Support expired. %s, %s", o.Family, o.Release)
-
-	case 12 < release && release < 14:
-		cmd += `| grep 'pkgs that look like they should be upgraded:' |
-			sed -e 's/pkgs that look like they should be upgraded://g'`
-
-	case 14 < release:
-		cmd += `| grep 'Packages that will be upgraded:' |
-			sed -e 's/Packages that will be upgraded://g'`
-
-	default:
-		return packNames, fmt.Errorf(
-			"Not supported yet. %s, %s", o.Family, o.Release)
+	if !found {
+		o.log.Debugf("Not found in meta: %s", current.Name)
+		err = cache.DB.EnsureBuckets(current)
+		if err != nil {
+			return fmt.Errorf("Failed to ensure buckets. err: %s", err)
+		}
+	} else {
+		if current.Distro.Family != old.Distro.Family ||
+			current.Distro.Release != old.Distro.Release {
+			o.log.Debugf("Need to refesh meta: %s", current.Name)
+			err = cache.DB.EnsureBuckets(current)
+			if err != nil {
+				return fmt.Errorf("Failed to ensure buckets. err: %s", err)
+			}
+		} else {
+			o.log.Debugf("Reuse meta: %s", current.Name)
+		}
 	}
 
+	if config.Conf.Debug {
+		cache.DB.PrettyPrint(current)
+	}
+	return nil
+}
+
+func (o *debian) fillCandidateVersion(before models.PackageInfoList) (filled []models.PackageInfo, err error) {
+	names := []string{}
+	for _, p := range before {
+		names = append(names, p.Name)
+	}
+	cmd := fmt.Sprintf("LANGUAGE=en_US.UTF-8 apt-cache policy %s", strings.Join(names, " "))
 	r := o.ssh(cmd, sudo)
-	if r.isSuccess(0, 1) {
-		packNames = strings.Split(strings.TrimSpace(r.Stdout), " ")
-		return packNames, nil
+	if !r.isSuccess() {
+		return nil, fmt.Errorf("Failed to SSH: %s", r)
 	}
-
-	return packNames, fmt.Errorf(
-		"Failed to %s. status: %d, stdout: %s, stderr: %s",
-		cmd, r.ExitStatus, r.Stdout, r.Stderr)
+	packChangelog := o.splitAptCachePolicy(r.Stdout)
+	for k, v := range packChangelog {
+		ver, err := o.parseAptCachePolicy(v, k)
+		if err != nil {
+			return nil, fmt.Errorf("Failed to parse %s", err)
+		}
+		p, found := before.FindByName(k)
+		if !found {
+			return nil, fmt.Errorf("Not found: %s", k)
+		}
+		p.NewVersion = ver.Candidate
+		filled = append(filled, p)
+	}
+	return
 }
 
 func (o *debian) GetUpgradablePackNames() (packNames []string, err error) {
-	cmd := util.PrependProxyEnv("LANG=en_US.UTF-8 apt-get upgrade --dry-run")
+	cmd := util.PrependProxyEnv("LANGUAGE=en_US.UTF-8 apt-get upgrade --dry-run")
 	r := o.ssh(cmd, sudo)
 	if r.isSuccess(0, 1) {
 		return o.parseAptGetUpgrade(r.Stdout)
@@ -404,8 +360,8 @@ func (o *debian) GetUpgradablePackNames() (packNames []string, err error) {
 }
 
 func (o *debian) parseAptGetUpgrade(stdout string) (upgradableNames []string, err error) {
-	startRe, _ := regexp.Compile(`The following packages will be upgraded:`)
-	stopRe, _ := regexp.Compile(`^(\d+) upgraded.*`)
+	startRe := regexp.MustCompile(`The following packages will be upgraded:`)
+	stopRe := regexp.MustCompile(`^(\d+) upgraded.*`)
 	startLineFound, stopLineFound := false, false
 
 	lines := strings.Split(stdout, "\n")
@@ -445,24 +401,26 @@ func (o *debian) parseAptGetUpgrade(stdout string) (upgradableNames []string, er
 	return
 }
 
-func (o *debian) scanPackageCveInfos(unsecurePacks []models.PackageInfo) (cvePacksList CvePacksList, err error) {
-
-	// { CVE ID: [packageInfo] }
-	cvePackages := make(map[string][]models.PackageInfo)
+func (o *debian) scanVulnInfos(upgradablePacks []models.PackageInfo) (models.VulnInfos, error) {
+	meta := cache.Meta{
+		Name:   o.getServerInfo().GetServerName(),
+		Distro: o.getServerInfo().Distro,
+		Packs:  upgradablePacks,
+	}
 
 	type strarray []string
 	resChan := make(chan struct {
 		models.PackageInfo
 		strarray
-	}, len(unsecurePacks))
-	errChan := make(chan error, len(unsecurePacks))
-	reqChan := make(chan models.PackageInfo, len(unsecurePacks))
+	}, len(upgradablePacks))
+	errChan := make(chan error, len(upgradablePacks))
+	reqChan := make(chan models.PackageInfo, len(upgradablePacks))
 	defer close(resChan)
 	defer close(errChan)
 	defer close(reqChan)
 
 	go func() {
-		for _, pack := range unsecurePacks {
+		for _, pack := range upgradablePacks {
 			reqChan <- pack
 		}
 	}()
@@ -470,11 +428,23 @@ func (o *debian) scanPackageCveInfos(unsecurePacks []models.PackageInfo) (cvePac
 	timeout := time.After(30 * 60 * time.Second)
 	concurrency := 10
 	tasks := util.GenWorkers(concurrency)
-	for range unsecurePacks {
+	for range upgradablePacks {
 		tasks <- func() {
 			select {
 			case pack := <-reqChan:
 				func(p models.PackageInfo) {
+					changelog := o.getChangelogCache(meta, p)
+					if 0 < len(changelog) {
+						cveIDs := o.getCveIDFromChangelog(changelog, p.Name, p.Version)
+						resChan <- struct {
+							models.PackageInfo
+							strarray
+						}{p, cveIDs}
+						return
+					}
+
+					// if the changelog is not in cache or failed to get from local cache,
+					// get the changelog of the package via internet.
 					if cveIDs, err := o.scanPackageCveIDs(p); err != nil {
 						errChan <- err
 					} else {
@@ -488,8 +458,10 @@ func (o *debian) scanPackageCveInfos(unsecurePacks []models.PackageInfo) (cvePac
 		}
 	}
 
+	// { CVE ID: [packageInfo] }
+	cvePackages := make(map[string][]models.PackageInfo)
 	errs := []error{}
-	for i := 0; i < len(unsecurePacks); i++ {
+	for i := 0; i < len(upgradablePacks); i++ {
 		select {
 		case pair := <-resChan:
 			pack := pair.PackageInfo
@@ -498,15 +470,13 @@ func (o *debian) scanPackageCveInfos(unsecurePacks []models.PackageInfo) (cvePac
 				cvePackages[cveID] = appendPackIfMissing(cvePackages[cveID], pack)
 			}
 			o.log.Infof("(%d/%d) Scanned %s-%s : %s",
-				i+1, len(unsecurePacks), pair.Name, pair.PackageInfo.Version, cveIDs)
+				i+1, len(upgradablePacks), pair.Name, pair.PackageInfo.Version, cveIDs)
 		case err := <-errChan:
 			errs = append(errs, err)
 		case <-timeout:
-			//TODO append to errs
-			return nil, fmt.Errorf("Timeout scanPackageCveIDs")
+			errs = append(errs, fmt.Errorf("Timeout scanPackageCveIDs"))
 		}
 	}
-
 	if 0 < len(errs) {
 		return nil, fmt.Errorf("%v", errs)
 	}
@@ -515,30 +485,43 @@ func (o *debian) scanPackageCveInfos(unsecurePacks []models.PackageInfo) (cvePac
 	for k := range cvePackages {
 		cveIDs = append(cveIDs, k)
 	}
-
 	o.log.Debugf("%d Cves are found. cves: %v", len(cveIDs), cveIDs)
-
-	o.log.Info("Fetching CVE details...")
-	cveDetails, err := cveapi.CveClient.FetchCveDetails(cveIDs)
-	if err != nil {
-		return nil, err
-	}
-	o.log.Info("Done")
-
-	for _, detail := range cveDetails {
-		cvePacksList = append(cvePacksList, CvePacksInfo{
-			CveID:     detail.CveID,
-			CveDetail: detail,
-			Packs:     cvePackages[detail.CveID],
-			//  CvssScore: cinfo.CvssScore(conf.Lang),
+	var vinfos models.VulnInfos
+	for k, v := range cvePackages {
+		vinfos = append(vinfos, models.VulnInfo{
+			CveID:    k,
+			Packages: v,
 		})
 	}
-	return
+	return vinfos, nil
+}
+
+func (o *debian) getChangelogCache(meta cache.Meta, pack models.PackageInfo) string {
+	cachedPack, found := meta.FindPack(pack.Name)
+	if !found {
+		return ""
+	}
+	if cachedPack.NewVersion != pack.NewVersion {
+		return ""
+	}
+	changelog, err := cache.DB.GetChangelog(meta.Name, pack.Name)
+	if err != nil {
+		o.log.Warnf("Failed to get changelog. bucket: %s, key:%s, err: %s",
+			meta.Name, pack.Name, err)
+		return ""
+	}
+	if len(changelog) == 0 {
+		return ""
+	}
+
+	o.log.Debugf("Cache hit: %s, len: %d, %s...",
+		meta.Name, len(changelog), util.Truncate(changelog, 30))
+	return changelog
 }
 
 func (o *debian) scanPackageCveIDs(pack models.PackageInfo) ([]string, error) {
 	cmd := ""
-	switch o.Family {
+	switch o.Distro.Family {
 	case "ubuntu":
 		cmd = fmt.Sprintf(`apt-get changelog %s | grep '\(urgency\|CVE\)'`, pack.Name)
 	case "debian":
@@ -551,37 +534,42 @@ func (o *debian) scanPackageCveIDs(pack models.PackageInfo) ([]string, error) {
 		o.log.Warnf("Failed to SSH: %s", r)
 		// Ignore this Error.
 		return nil, nil
-
 	}
-	return o.getCveIDParsingChangelog(r.Stdout, pack.Name, pack.Version)
+
+	if 0 < len(strings.TrimSpace(r.Stdout)) {
+		err := cache.DB.PutChangelog(o.getServerInfo().GetServerName(), pack.Name, r.Stdout)
+		if err != nil {
+			return nil, fmt.Errorf("Failed to put changelog into cache")
+		}
+	}
+	// No error will be returned. Only logging.
+	return o.getCveIDFromChangelog(r.Stdout, pack.Name, pack.Version), nil
 }
 
-func (o *debian) getCveIDParsingChangelog(changelog string,
-	packName string, versionOrLater string) (cveIDs []string, err error) {
+func (o *debian) getCveIDFromChangelog(changelog string,
+	packName string, versionOrLater string) []string {
 
-	cveIDs, err = o.parseChangelog(changelog, packName, versionOrLater)
-	if err == nil {
-		return
+	if cveIDs, err := o.parseChangelog(changelog, packName, versionOrLater); err == nil {
+		return cveIDs
 	}
 
 	ver := strings.Split(versionOrLater, "ubuntu")[0]
-	cveIDs, err = o.parseChangelog(changelog, packName, ver)
-	if err == nil {
-		return
+	if cveIDs, err := o.parseChangelog(changelog, packName, ver); err == nil {
+		return cveIDs
 	}
 
 	splittedByColon := strings.Split(versionOrLater, ":")
 	if 1 < len(splittedByColon) {
 		ver = splittedByColon[1]
 	}
-	cveIDs, err = o.parseChangelog(changelog, packName, ver)
+	cveIDs, err := o.parseChangelog(changelog, packName, ver)
 	if err == nil {
-		return
+		return cveIDs
 	}
 
-	//TODO report as unable to parse changelog.
+	// Only logging the error.
 	o.log.Error(err)
-	return []string{}, nil
+	return []string{}
 }
 
 // Collect CVE-IDs included in the changelog.
@@ -589,8 +577,8 @@ func (o *debian) getCveIDParsingChangelog(changelog string,
 func (o *debian) parseChangelog(changelog string,
 	packName string, versionOrLater string) (cveIDs []string, err error) {
 
-	cveRe, _ := regexp.Compile(`(CVE-\d{4}-\d{4})`)
-	stopRe, _ := regexp.Compile(fmt.Sprintf(`\(%s\)`, regexp.QuoteMeta(versionOrLater)))
+	cveRe := regexp.MustCompile(`(CVE-\d{4}-\d{4,})`)
+	stopRe := regexp.MustCompile(fmt.Sprintf(`\(%s\)`, regexp.QuoteMeta(versionOrLater)))
 	stopLineFound := false
 	lines := strings.Split(changelog, "\n")
 	for _, line := range lines {
@@ -598,7 +586,7 @@ func (o *debian) parseChangelog(changelog string,
 			o.log.Debugf("Found the stop line. line: %s", line)
 			stopLineFound = true
 			break
-		} else if matches := cveRe.FindAllString(line, -1); len(matches) > 0 {
+		} else if matches := cveRe.FindAllString(line, -1); 0 < len(matches) {
 			for _, m := range matches {
 				cveIDs = util.AppendIfMissing(cveIDs, m)
 			}
@@ -614,6 +602,29 @@ func (o *debian) parseChangelog(changelog string,
 	return
 }
 
+func (o *debian) splitAptCachePolicy(stdout string) map[string]string {
+	//  re := regexp.MustCompile(`(?m:^[^ \t]+:$)`)
+	re := regexp.MustCompile(`(?m:^[^ \t]+:\r\n)`)
+	ii := re.FindAllStringIndex(stdout, -1)
+	ri := []int{}
+	for i := len(ii) - 1; 0 <= i; i-- {
+		ri = append(ri, ii[i][0])
+	}
+	splitted := []string{}
+	lasti := len(stdout)
+	for _, i := range ri {
+		splitted = append(splitted, stdout[i:lasti])
+		lasti = i
+	}
+
+	packChangelog := map[string]string{}
+	for _, r := range splitted {
+		packName := r[:strings.Index(r, ":")]
+		packChangelog[packName] = r
+	}
+	return packChangelog
+}
+
 type packCandidateVer struct {
 	Name      string
 	Installed string

scan/debian_test.go

@@ -18,10 +18,14 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
 package scan
 
 import (
+	"os"
 	"reflect"
 	"testing"
 
+	"github.com/Sirupsen/logrus"
+	"github.com/future-architect/vuls/cache"
 	"github.com/future-architect/vuls/config"
+	"github.com/future-architect/vuls/models"
 	"github.com/k0kubun/pp"
 )
 
@@ -54,7 +58,7 @@ func TestParseScannedPackagesLineDebian(t *testing.T) {
 
 }
 
-func TestgetCveIDParsingChangelog(t *testing.T) {
+func TestGetCveIDParsingChangelog(t *testing.T) {
 
 	var tests = []struct {
 		in       []string
@@ -86,12 +90,11 @@ systemd (227-1) unstable; urgency=medium`,
 				"CVE-2015-3210",
 			},
 		},
-
 		{
 			// ver
 			[]string{
 				"libpcre3",
-				"2:8.38-1ubuntu1",
+				"2:8.35-7.1ubuntu1",
 				`pcre3 (2:8.38-2) unstable; urgency=low
 pcre3 (2:8.38-1) unstable; urgency=low
 pcre3 (2:8.35-8) unstable; urgency=low
@@ -110,7 +113,6 @@ pcre3 (2:8.35-7) unstable; urgency=medium`,
 				"CVE-2015-3210",
 			},
 		},
-
 		{
 			// ver-ubuntu3
 			[]string{
@@ -151,7 +153,7 @@ sysvinit (2.88dsf-57) unstable; urgency=low`,
 util-linux (2.27.1-3) unstable; urgency=medium
 CVE-2015-2325: heap buffer overflow in compile_branch(). (Closes: #781795)
 CVE-2015-2326: heap buffer overflow in pcre_compile2(). (Closes: #783285)
-CVE-2015-3210: heap buffer overflow in pcre_compile2() /
+CVE-2015-3210: CVE-2016-1000000heap buffer overflow in pcre_compile2() /
 util-linux (2.27.1-2) unstable; urgency=medium
 util-linux (2.27.1-1ubuntu4) xenial; urgency=medium
 util-linux (2.27.1-1ubuntu3) xenial; urgency=medium
@@ -178,15 +180,17 @@ util-linux (2.26.2-6) unstable; urgency=medium`,
 				"CVE-2015-2325",
 				"CVE-2015-2326",
 				"CVE-2015-3210",
+				"CVE-2016-1000000",
 			},
 		},
 	}
 
 	d := newDebian(config.ServerInfo{})
 	for _, tt := range tests {
-		actual, _ := d.getCveIDParsingChangelog(tt.in[2], tt.in[0], tt.in[1])
+		actual := d.getCveIDFromChangelog(tt.in[2], tt.in[0], tt.in[1])
 		if len(actual) != len(tt.expected) {
 			t.Errorf("Len of return array are'nt same. expected %#v, actual %#v", tt.expected, actual)
+			t.Errorf(pp.Sprintf("%s", tt.in))
 			continue
 		}
 		for i := range tt.expected {
@@ -195,13 +199,6 @@ util-linux (2.26.2-6) unstable; urgency=medium`,
 			}
 		}
 	}
-
-	for _, tt := range tests {
-		_, err := d.getCveIDParsingChangelog(tt.in[2], tt.in[0], "version number do'nt match case")
-		if err != nil {
-			t.Errorf("Returning error is unexpected")
-		}
-	}
 }
 
 func TestGetUpdatablePackNames(t *testing.T) {
@@ -520,6 +517,95 @@ Calculating upgrade... Done
 	}
 }
 
+func TestGetChangelogCache(t *testing.T) {
+	const servername = "server1"
+	pack := models.PackageInfo{
+		Name:       "apt",
+		Version:    "1.0.0",
+		NewVersion: "1.0.1",
+	}
+	var meta = cache.Meta{
+		Name: servername,
+		Distro: config.Distro{
+			Family:  "ubuntu",
+			Release: "16.04",
+		},
+		Packs: []models.PackageInfo{pack},
+	}
+
+	const path = "/tmp/vuls-test-cache-11111111.db"
+	log := logrus.NewEntry(&logrus.Logger{})
+	if err := cache.SetupBolt(path, log); err != nil {
+		t.Errorf("Failed to setup bolt: %s", err)
+	}
+	defer os.Remove(path)
+
+	if err := cache.DB.EnsureBuckets(meta); err != nil {
+		t.Errorf("Failed to ensure buckets: %s", err)
+	}
+
+	d := newDebian(config.ServerInfo{})
+	actual := d.getChangelogCache(meta, pack)
+	if actual != "" {
+		t.Errorf("Failed to get empty stirng from cache:")
+	}
+
+	clog := "changelog-text"
+	if err := cache.DB.PutChangelog(servername, "apt", clog); err != nil {
+		t.Errorf("Failed to put changelog: %s", err)
+	}
+
+	actual = d.getChangelogCache(meta, pack)
+	if actual != clog {
+		t.Errorf("Failed to get changelog from cache: %s", actual)
+	}
+
+	// increment a version of the pack
+	pack.NewVersion = "1.0.2"
+	actual = d.getChangelogCache(meta, pack)
+	if actual != "" {
+		t.Errorf("The changelog is not invalidated: %s", actual)
+	}
+
+	// change a name of the pack
+	pack.Name = "bash"
+	actual = d.getChangelogCache(meta, pack)
+	if actual != "" {
+		t.Errorf("The changelog is not invalidated: %s", actual)
+	}
+}
+
+func TestSplitAptCachePolicy(t *testing.T) {
+	var tests = []struct {
+		stdout   string
+		expected map[string]string
+	}{
+		// This function parse apt-cache policy by using Regexp multi-line mode.
+		// So, test data includes "\r\n"
+		{
+			"apt:\r\n  Installed: 1.2.6\r\n  Candidate: 1.2.12~ubuntu16.04.1\r\n  Version table:\r\n     1.2.12~ubuntu16.04.1 500\r\n        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages\r\n     1.2.10ubuntu1 500\r\n        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages\r\n *** 1.2.6 100\r\n        100 /var/lib/dpkg/status\r\napt-utils:\r\n  Installed: 1.2.6\r\n  Candidate: 1.2.12~ubuntu16.04.1\r\n  Version table:\r\n     1.2.12~ubuntu16.04.1 500\r\n        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages\r\n     1.2.10ubuntu1 500\r\n        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages\r\n *** 1.2.6 100\r\n        100 /var/lib/dpkg/status\r\nbase-files:\r\n  Installed: 9.4ubuntu3\r\n  Candidate: 9.4ubuntu4.2\r\n  Version table:\r\n     9.4ubuntu4.2 500\r\n        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages\r\n     9.4ubuntu4 500\r\n        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages\r\n *** 9.4ubuntu3 100\r\n        100 /var/lib/dpkg/status\r\n",
+
+			map[string]string{
+				"apt": "apt:\r\n  Installed: 1.2.6\r\n  Candidate: 1.2.12~ubuntu16.04.1\r\n  Version table:\r\n     1.2.12~ubuntu16.04.1 500\r\n        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages\r\n     1.2.10ubuntu1 500\r\n        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages\r\n *** 1.2.6 100\r\n        100 /var/lib/dpkg/status\r\n",
+
+				"apt-utils": "apt-utils:\r\n  Installed: 1.2.6\r\n  Candidate: 1.2.12~ubuntu16.04.1\r\n  Version table:\r\n     1.2.12~ubuntu16.04.1 500\r\n        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages\r\n     1.2.10ubuntu1 500\r\n        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages\r\n *** 1.2.6 100\r\n        100 /var/lib/dpkg/status\r\n",
+
+				"base-files": "base-files:\r\n  Installed: 9.4ubuntu3\r\n  Candidate: 9.4ubuntu4.2\r\n  Version table:\r\n     9.4ubuntu4.2 500\r\n        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages\r\n     9.4ubuntu4 500\r\n        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages\r\n *** 9.4ubuntu3 100\r\n        100 /var/lib/dpkg/status\r\n",
+			},
+		},
+	}
+
+	d := newDebian(config.ServerInfo{})
+	for _, tt := range tests {
+		actual := d.splitAptCachePolicy(tt.stdout)
+		if !reflect.DeepEqual(tt.expected, actual) {
+			e := pp.Sprintf("%v", tt.expected)
+			a := pp.Sprintf("%v", actual)
+			t.Errorf("expected %s, actual %s", e, a)
+		}
+	}
+}
+
 func TestParseAptCachePolicy(t *testing.T) {
 
 	var tests = []struct {

scan/freebsd.go

@@ -1,3 +1,20 @@
+/* Vuls - Vulnerability Scanner
+Copyright (C) 2016  Future Architect, Inc. Japan.
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
 package scan
 
 import (
@@ -5,7 +22,6 @@ import (
 	"strings"
 
 	"github.com/future-architect/vuls/config"
-	"github.com/future-architect/vuls/cveapi"
 	"github.com/future-architect/vuls/models"
 	"github.com/future-architect/vuls/util"
 )
@@ -19,18 +35,22 @@ type bsd struct {
 func newBsd(c config.ServerInfo) *bsd {
 	d := &bsd{}
 	d.log = util.NewCustomLogger(c)
+	d.setServerInfo(c)
 	return d
 }
 
 //https://github.com/mizzy/specinfra/blob/master/lib/specinfra/helper/detect_os/freebsd.rb
 func detectFreebsd(c config.ServerInfo) (itsMe bool, bsd osTypeInterface) {
 	bsd = newBsd(c)
-	c.Family = "FreeBSD"
+
+	// Prevent from adding `set -o pipefail` option
+	c.Distro = config.Distro{Family: "FreeBSD"}
+
 	if r := sshExec(c, "uname", noSudo); r.isSuccess() {
 		if strings.Contains(r.Stdout, "FreeBSD") == true {
 			if b := sshExec(c, "uname -r", noSudo); b.isSuccess() {
-				bsd.setDistributionInfo("FreeBSD", strings.TrimSpace(b.Stdout))
-				bsd.setServerInfo(c)
+				rel := strings.TrimSpace(b.Stdout)
+				bsd.setDistro("FreeBSD", rel)
 				return true, bsd
 			}
 		}
@@ -45,6 +65,10 @@ func (o *bsd) checkIfSudoNoPasswd() error {
 	return nil
 }
 
+func (o *bsd) checkDependencies() error {
+	return nil
+}
+
 func (o *bsd) install() error {
 	return nil
 }
@@ -62,12 +86,12 @@ func (o *bsd) scanPackages() error {
 	}
 	o.setPackages(packs)
 
-	var unsecurePacks []CvePacksInfo
-	if unsecurePacks, err = o.scanUnsecurePackages(); err != nil {
+	var vinfos []models.VulnInfo
+	if vinfos, err = o.scanUnsecurePackages(); err != nil {
 		o.log.Errorf("Failed to scan vulnerable packages")
 		return err
 	}
-	o.setUnsecurePackages(unsecurePacks)
+	o.setVulnInfos(vinfos)
 	return nil
 }
 
@@ -80,7 +104,7 @@ func (o *bsd) scanInstalledPackages() ([]models.PackageInfo, error) {
 	return o.parsePkgVersion(r.Stdout), nil
 }
 
-func (o *bsd) scanUnsecurePackages() (cvePacksList []CvePacksInfo, err error) {
+func (o *bsd) scanUnsecurePackages() (vulnInfos []models.VulnInfo, err error) {
 	const vulndbPath = "/tmp/vuln.db"
 	cmd := "rm -f " + vulndbPath
 	r := o.ssh(cmd, noSudo)
@@ -95,7 +119,7 @@ func (o *bsd) scanUnsecurePackages() (cvePacksList []CvePacksInfo, err error) {
 	}
 	if r.ExitStatus == 0 {
 		// no vulnerabilities
-		return []CvePacksInfo{}, nil
+		return []models.VulnInfo{}, nil
 	}
 
 	var packAdtRslt []pkgAuditResult
@@ -126,34 +150,22 @@ func (o *bsd) scanUnsecurePackages() (cvePacksList []CvePacksInfo, err error) {
 		}
 	}
 
-	cveIDs := []string{}
 	for k := range cveIDAdtMap {
-		cveIDs = append(cveIDs, k)
-	}
-
-	cveDetails, err := cveapi.CveClient.FetchCveDetails(cveIDs)
-	if err != nil {
-		return nil, err
-	}
-	o.log.Info("Done")
-
-	for _, d := range cveDetails {
 		packs := []models.PackageInfo{}
-		for _, r := range cveIDAdtMap[d.CveID] {
+		for _, r := range cveIDAdtMap[k] {
 			packs = append(packs, r.pack)
 		}
 
 		disAdvs := []models.DistroAdvisory{}
-		for _, r := range cveIDAdtMap[d.CveID] {
+		for _, r := range cveIDAdtMap[k] {
 			disAdvs = append(disAdvs, models.DistroAdvisory{
 				AdvisoryID: r.vulnIDCveIDs.vulnID,
 			})
 		}
 
-		cvePacksList = append(cvePacksList, CvePacksInfo{
-			CveID:            d.CveID,
-			CveDetail:        d,
-			Packs:            packs,
+		vulnInfos = append(vulnInfos, models.VulnInfo{
+			CveID:            k,
+			Packages:         packs,
 			DistroAdvisories: disAdvs,
 		})
 	}

scan/freebsd_test.go

@@ -139,17 +139,17 @@ WWW: https://vuxml.FreeBSD.org/freebsd/ab3e98d9-8175-11e4-907d-d050992ecde8.html
 
 	d := newBsd(config.ServerInfo{})
 	for _, tt := range tests {
-		aName, aCveIDs, aVunlnID := d.parseBlock(tt.in)
+		aName, aCveIDs, aVulnID := d.parseBlock(tt.in)
 		if tt.name != aName {
-			t.Errorf("expected vulnID: %s, actual %s", tt.vulnID, aVunlnID)
+			t.Errorf("expected vulnID: %s, actual %s", tt.vulnID, aVulnID)
 		}
 		for i := range tt.cveIDs {
 			if tt.cveIDs[i] != aCveIDs[i] {
 				t.Errorf("expected cveID: %s, actual %s", tt.cveIDs[i], aCveIDs[i])
 			}
 		}
-		if tt.vulnID != aVunlnID {
-			t.Errorf("expected vulnID: %s, actual %s", tt.vulnID, aVunlnID)
+		if tt.vulnID != aVulnID {
+			t.Errorf("expected vulnID: %s, actual %s", tt.vulnID, aVulnID)
 		}
 	}
 }

scan/redhat.go

@@ -26,7 +26,6 @@ import (
 	"time"
 
 	"github.com/future-architect/vuls/config"
-	"github.com/future-architect/vuls/cveapi"
 	"github.com/future-architect/vuls/models"
 	"github.com/future-architect/vuls/util"
 
@@ -42,16 +41,16 @@ type redhat struct {
 func newRedhat(c config.ServerInfo) *redhat {
 	r := &redhat{}
 	r.log = util.NewCustomLogger(c)
+	r.setServerInfo(c)
 	return r
 }
 
 // https://github.com/serverspec/specinfra/blob/master/lib/specinfra/helper/detect_os/redhat.rb
 func detectRedhat(c config.ServerInfo) (itsMe bool, red osTypeInterface) {
 	red = newRedhat(c)
-	red.setServerInfo(c)
 
 	if r := sshExec(c, "ls /etc/fedora-release", noSudo); r.isSuccess() {
-		red.setDistributionInfo("fedora", "unknown")
+		red.setDistro("fedora", "unknown")
 		Log.Warn("Fedora not tested yet: %s", r)
 		return true, red
 	}
@@ -62,7 +61,7 @@ func detectRedhat(c config.ServerInfo) (itsMe bool, red osTypeInterface) {
 		// $ cat /etc/redhat-release
 		// CentOS release 6.5 (Final)
 		if r := sshExec(c, "cat /etc/redhat-release", noSudo); r.isSuccess() {
-			re, _ := regexp.Compile(`(.*) release (\d[\d.]*)`)
+			re := regexp.MustCompile(`(.*) release (\d[\d.]*)`)
 			result := re.FindStringSubmatch(strings.TrimSpace(r.Stdout))
 			if len(result) != 3 {
 				Log.Warn("Failed to parse RedHat/CentOS version: %s", r)
@@ -72,9 +71,9 @@ func detectRedhat(c config.ServerInfo) (itsMe bool, red osTypeInterface) {
 			release := result[2]
 			switch strings.ToLower(result[1]) {
 			case "centos", "centos linux":
-				red.setDistributionInfo("centos", release)
+				red.setDistro("centos", release)
 			default:
-				red.setDistributionInfo("rhel", release)
+				red.setDistro("rhel", release)
 			}
 			return true, red
 		}
@@ -90,7 +89,7 @@ func detectRedhat(c config.ServerInfo) (itsMe bool, red osTypeInterface) {
 				release = fields[4]
 			}
 		}
-		red.setDistributionInfo(family, release)
+		red.setDistro(family, release)
 		return true, red
 	}
 
@@ -99,7 +98,7 @@ func detectRedhat(c config.ServerInfo) (itsMe bool, red osTypeInterface) {
 }
 
 func (o *redhat) checkIfSudoNoPasswd() error {
-	r := o.ssh("yum --version", sudo)
+	r := o.ssh("yum --version", o.sudo())
 	if !r.isSuccess() {
 		o.log.Errorf("sudo error on %s", r)
 		return fmt.Errorf("Failed to sudo: %s", r)
@@ -108,104 +107,66 @@ func (o *redhat) checkIfSudoNoPasswd() error {
 	return nil
 }
 
-// CentOS 5 ... yum-plugin-security, yum-changelog
-// CentOS 6 ... yum-plugin-security, yum-plugin-changelog
-// CentOS 7 ... yum-plugin-security, yum-plugin-changelog
+// CentOS 5 ... yum-changelog
+// CentOS 6 ... yum-plugin-changelog
+// CentOS 7 ... yum-plugin-changelog
 // RHEL, Amazon ... no additinal packages needed
-func (o *redhat) install() error {
-
-	switch o.Family {
+func (o *redhat) checkDependencies() error {
+	switch o.Distro.Family {
 	case "rhel", "amazon":
-		o.log.Infof("Nothing to do")
+		//  o.log.Infof("Nothing to do")
 		return nil
-	}
 
-	if err := o.installYumPluginSecurity(); err != nil {
-		return err
-	}
-	return o.installYumChangelog()
-}
-
-func (o *redhat) installYumPluginSecurity() error {
-
-	if r := o.ssh("rpm -q yum-plugin-security", noSudo); r.isSuccess() {
-		o.log.Infof("Ignored: yum-plugin-security already installed")
-		return nil
-	}
-
-	o.log.Info("Installing yum-plugin-security...")
-	cmd := util.PrependProxyEnv("yum install -y yum-plugin-security")
-	if r := o.ssh(cmd, sudo); !r.isSuccess() {
-		return fmt.Errorf("Failed to SSH: %s", r)
-	}
-	return nil
-}
-
-func (o *redhat) installYumChangelog() error {
-
-	if o.Family == "centos" {
+	case "centos":
 		var majorVersion int
-		if 0 < len(o.Release) {
-			majorVersion, _ = strconv.Atoi(strings.Split(o.Release, ".")[0])
+		if 0 < len(o.Distro.Release) {
+			majorVersion, _ = strconv.Atoi(strings.Split(o.Distro.Release, ".")[0])
 		} else {
-			return fmt.Errorf(
-				"Not implemented yet. family: %s, release: %s",
-				o.Family, o.Release)
+			return fmt.Errorf("Not implemented yet: %s", o.Distro)
 		}
 
-		var packName = ""
+		var name = "yum-plugin-changelog"
 		if majorVersion < 6 {
-			packName = "yum-changelog"
-		} else {
-			packName = "yum-plugin-changelog"
+			name = "yum-changelog"
 		}
 
-		cmd := "rpm -q " + packName
+		cmd := "rpm -q " + name
 		if r := o.ssh(cmd, noSudo); r.isSuccess() {
-			o.log.Infof("Ignored: %s already installed", packName)
 			return nil
 		}
+		o.lackDependencies = []string{name}
+		return nil
 
-		o.log.Infof("Installing %s...", packName)
-		cmd = util.PrependProxyEnv("yum install -y " + packName)
+	default:
+		return fmt.Errorf("Not implemented yet: %s", o.Distro)
+	}
+}
+
+func (o *redhat) install() error {
+	for _, name := range o.lackDependencies {
+		cmd := util.PrependProxyEnv("yum install -y " + name)
 		if r := o.ssh(cmd, sudo); !r.isSuccess() {
 			return fmt.Errorf("Failed to SSH: %s", r)
 		}
-		o.log.Infof("Installed: %s", packName)
+		o.log.Infof("Installed: %s", name)
 	}
 	return nil
 }
 
 func (o *redhat) checkRequiredPackagesInstalled() error {
-	if config.Conf.UseYumPluginSecurity {
-		// check if yum-plugin-security is installed.
-		// Amazon Linux, REHL can execute 'yum updateinfo --security updates' without yum-plugin-security
-		if o.Family == "centos" {
-			cmd := "rpm -q yum-plugin-security"
-			if r := o.ssh(cmd, noSudo); !r.isSuccess() {
-				msg := "yum-plugin-security is not installed"
-				o.log.Errorf(msg)
-				return fmt.Errorf(msg)
-			}
-		}
-		return nil
-	}
-
-	if o.Family == "centos" {
+	if o.Distro.Family == "centos" {
 		var majorVersion int
-		if 0 < len(o.Release) {
-			majorVersion, _ = strconv.Atoi(strings.Split(o.Release, ".")[0])
+		if 0 < len(o.Distro.Release) {
+			majorVersion, _ = strconv.Atoi(strings.Split(o.Distro.Release, ".")[0])
 		} else {
-			msg := fmt.Sprintf("Not implemented yet. family: %s, release: %s", o.Family, o.Release)
+			msg := fmt.Sprintf("Not implemented yet: %s", o.Distro)
 			o.log.Errorf(msg)
 			return fmt.Errorf(msg)
 		}
 
-		var packName = ""
+		var packName = "yum-plugin-changelog"
 		if majorVersion < 6 {
 			packName = "yum-changelog"
-		} else {
-			packName = "yum-plugin-changelog"
 		}
 
 		cmd := "rpm -q " + packName
@@ -227,12 +188,12 @@ func (o *redhat) scanPackages() error {
 	}
 	o.setPackages(packs)
 
-	var unsecurePacks []CvePacksInfo
-	if unsecurePacks, err = o.scanUnsecurePackages(); err != nil {
+	var vinfos []models.VulnInfo
+	if vinfos, err = o.scanVulnInfos(); err != nil {
 		o.log.Errorf("Failed to scan vulnerable packages")
 		return err
 	}
-	o.setUnsecurePackages(unsecurePacks)
+	o.setVulnInfos(vinfos)
 	return nil
 }
 
@@ -273,8 +234,8 @@ func (o *redhat) parseScannedPackagesLine(line string) (models.PackageInfo, erro
 	}, nil
 }
 
-func (o *redhat) scanUnsecurePackages() ([]CvePacksInfo, error) {
-	if o.Family != "centos" || config.Conf.UseYumPluginSecurity {
+func (o *redhat) scanVulnInfos() ([]models.VulnInfo, error) {
+	if o.Distro.Family != "centos" {
 		// Amazon, RHEL has yum updateinfo as default
 		// yum updateinfo can collenct vendor advisory information.
 		return o.scanUnsecurePackagesUsingYumPluginSecurity()
@@ -284,10 +245,15 @@ func (o *redhat) scanUnsecurePackages() ([]CvePacksInfo, error) {
 	return o.scanUnsecurePackagesUsingYumCheckUpdate()
 }
 
-//TODO return whether already expired.
-func (o *redhat) scanUnsecurePackagesUsingYumCheckUpdate() (CvePacksList, error) {
+// For CentOS
+func (o *redhat) scanUnsecurePackagesUsingYumCheckUpdate() (models.VulnInfos, error) {
+	cmd := "LANGUAGE=en_US.UTF-8 yum --color=never %s check-update"
+	if o.getServerInfo().Enablerepo != "" {
+		cmd = fmt.Sprintf(cmd, "--enablerepo="+o.getServerInfo().Enablerepo)
+	} else {
+		cmd = fmt.Sprintf(cmd, "")
+	}
 
-	cmd := "LANG=en_US.UTF-8 yum --color=never check-update"
 	r := o.ssh(util.PrependProxyEnv(cmd), sudo)
 	if !r.isSuccess(0, 100) {
 		//returns an exit code of 100 if there are available updates.
@@ -301,19 +267,23 @@ func (o *redhat) scanUnsecurePackagesUsingYumCheckUpdate() (CvePacksList, error)
 	}
 	o.log.Debugf("%s", pp.Sprintf("%v", packInfoList))
 
+	// set candidate version info
+	o.Packages.MergeNewVersion(packInfoList)
+
 	// Collect CVE-IDs in changelog
 	type PackInfoCveIDs struct {
 		PackInfo models.PackageInfo
 		CveIDs   []string
 	}
 
-	// { packageName: changelog-lines }
-	var rpm2changelog map[string]*string
 	allChangelog, err := o.getAllChangelog(packInfoList)
 	if err != nil {
 		o.log.Errorf("Failed to getAllchangelog. err: %s", err)
 		return nil, err
 	}
+
+	// { packageName: changelog-lines }
+	var rpm2changelog map[string]*string
 	rpm2changelog, err = o.parseAllChangelog(allChangelog)
 	if err != nil {
 		return nil, fmt.Errorf("Failed to parseAllChangelog. err: %s", err)
@@ -370,39 +340,20 @@ func (o *redhat) scanUnsecurePackagesUsingYumCheckUpdate() (CvePacksList, error)
 	cveIDPackInfoMap := make(map[string][]models.PackageInfo)
 	for _, res := range results {
 		for _, cveID := range res.CveIDs {
-			//  packInfo, found := o.Packages.FindByName(res.Packname)
-			//  if !found {
-			//      return CvePacksList{}, fmt.Errorf(
-			//          "Faild to transform data structure: %v", res.Packname)
-			//  }
-			cveIDPackInfoMap[cveID] = append(cveIDPackInfoMap[cveID], res.PackInfo)
+			cveIDPackInfoMap[cveID] = append(
+				cveIDPackInfoMap[cveID], res.PackInfo)
 		}
 	}
 
-	var uniqueCveIDs []string
-	for cveID := range cveIDPackInfoMap {
-		uniqueCveIDs = append(uniqueCveIDs, cveID)
-	}
-
-	// cveIDs => []cve.CveInfo
-	o.log.Info("Fetching CVE details...")
-	cveDetails, err := cveapi.CveClient.FetchCveDetails(uniqueCveIDs)
-	if err != nil {
-		return nil, err
-	}
-	o.log.Info("Done")
-
-	cvePacksList := []CvePacksInfo{}
-	for _, detail := range cveDetails {
+	vinfos := []models.VulnInfo{}
+	for k, v := range cveIDPackInfoMap {
 		// Amazon, RHEL do not use this method, so VendorAdvisory do not set.
-		cvePacksList = append(cvePacksList, CvePacksInfo{
-			CveID:     detail.CveID,
-			CveDetail: detail,
-			Packs:     cveIDPackInfoMap[detail.CveID],
-			//  CvssScore: cinfo.CvssScore(conf.Lang),
+		vinfos = append(vinfos, models.VulnInfo{
+			CveID:    k,
+			Packages: v,
 		})
 	}
-	return cvePacksList, nil
+	return vinfos, nil
 }
 
 // parseYumCheckUpdateLines parse yum check-update to get package name, candidate version
@@ -416,7 +367,9 @@ func (o *redhat) parseYumCheckUpdateLines(stdout string) (results models.Package
 			continue
 		}
 		if needToParse {
-			if strings.HasPrefix(line, "Obsoleting") {
+			if strings.HasPrefix(line, "Obsoleting") ||
+				strings.HasPrefix(line, "Security:") {
+				// see https://github.com/future-architect/vuls/issues/165
 				continue
 			}
 			candidate, err := o.parseYumCheckUpdateLine(line)
@@ -433,6 +386,7 @@ func (o *redhat) parseYumCheckUpdateLines(stdout string) (results models.Package
 			}
 			installed.NewVersion = candidate.NewVersion
 			installed.NewRelease = candidate.NewRelease
+			installed.Repository = candidate.Repository
 			results = append(results, installed)
 		}
 	}
@@ -441,7 +395,7 @@ func (o *redhat) parseYumCheckUpdateLines(stdout string) (results models.Package
 
 func (o *redhat) parseYumCheckUpdateLine(line string) (models.PackageInfo, error) {
 	fields := strings.Fields(line)
-	if len(fields) != 3 {
+	if len(fields) < 3 {
 		return models.PackageInfo{}, fmt.Errorf("Unknown format: %s", line)
 	}
 	splitted := strings.Split(fields[0], ".")
@@ -452,16 +406,19 @@ func (o *redhat) parseYumCheckUpdateLine(line string) (models.PackageInfo, error
 		packName = strings.Join(strings.Split(fields[0], ".")[0:(len(splitted)-1)], ".")
 	}
 
-	fields = strings.Split(fields[1], "-")
-	if len(fields) != 2 {
+	verfields := strings.Split(fields[1], "-")
+	if len(verfields) != 2 {
 		return models.PackageInfo{}, fmt.Errorf("Unknown format: %s", line)
 	}
-	version := o.regexpReplace(fields[0], `^[0-9]+:`, "")
-	release := fields[1]
+	version := o.regexpReplace(verfields[0], `^[0-9]+:`, "")
+	release := verfields[1]
+	repos := strings.Join(fields[2:len(fields)], " ")
+
 	return models.PackageInfo{
 		Name:       packName,
 		NewVersion: version,
 		NewRelease: release,
+		Repository: repos,
 	}, nil
 }
 
@@ -471,18 +428,19 @@ func (o *redhat) mkPstring() *string {
 }
 
 func (o *redhat) regexpReplace(src string, pat string, rep string) string {
-	r := regexp.MustCompile(pat)
-	return r.ReplaceAllString(src, rep)
+	re := regexp.MustCompile(pat)
+	return re.ReplaceAllString(src, rep)
 }
 
+var changeLogCVEPattern = regexp.MustCompile(`CVE-[0-9]+-[0-9]+`)
+
 func (o *redhat) getChangelogCVELines(rpm2changelog map[string]*string, packInfo models.PackageInfo) string {
 	rpm := fmt.Sprintf("%s-%s-%s", packInfo.Name, packInfo.NewVersion, packInfo.NewRelease)
 	retLine := ""
 	if rpm2changelog[rpm] != nil {
 		lines := strings.Split(*rpm2changelog[rpm], "\n")
 		for _, line := range lines {
-			match, _ := regexp.MatchString("CVE-[0-9]+-[0-9]+", line)
-			if match {
+			if changeLogCVEPattern.MatchString(line) {
 				retLine += fmt.Sprintf("%s\n", line)
 			}
 		}
@@ -492,12 +450,10 @@ func (o *redhat) getChangelogCVELines(rpm2changelog map[string]*string, packInfo
 
 func (o *redhat) parseAllChangelog(allChangelog string) (map[string]*string, error) {
 	var majorVersion int
-	if 0 < len(o.Release) && o.Family == "centos" {
-		majorVersion, _ = strconv.Atoi(strings.Split(o.Release, ".")[0])
+	if 0 < len(o.Distro.Release) && o.Distro.Family == "centos" {
+		majorVersion, _ = strconv.Atoi(strings.Split(o.Distro.Release, ".")[0])
 	} else {
-		return nil, fmt.Errorf(
-			"Not implemented yet. family: %s, release: %s",
-			o.Family, o.Release)
+		return nil, fmt.Errorf("Not implemented yet: %s", o.getDistro())
 	}
 
 	orglines := strings.Split(allChangelog, "\n")
@@ -510,7 +466,7 @@ func (o *redhat) parseAllChangelog(allChangelog string) (map[string]*string, err
 			/* for CentOS5 (yum-util < 1.1.20) */
 			prev = false
 			now = false
-			if i > 0 {
+			if 0 < i {
 				prev, err = o.isRpmPackageNameLine(orglines[i-1])
 				if err != nil {
 					return nil, err
@@ -560,8 +516,7 @@ func (o *redhat) parseAllChangelog(allChangelog string) (map[string]*string, err
 				rpm2changelog[rpm] = pNewString
 			}
 		} else {
-			stop, _ := regexp.MatchString("^Dependencies Resolved", line)
-			if stop {
+			if strings.HasPrefix(line, "Dependencies Resolved") {
 				return rpm2changelog, nil
 			}
 			*writePointer += fmt.Sprintf("%s\n", line)
@@ -570,22 +525,27 @@ func (o *redhat) parseAllChangelog(allChangelog string) (map[string]*string, err
 	return rpm2changelog, nil
 }
 
+// CentOS
 func (o *redhat) getAllChangelog(packInfoList models.PackageInfoList) (stdout string, err error) {
 	packageNames := ""
 	for _, packInfo := range packInfoList {
 		packageNames += fmt.Sprintf("%s ", packInfo.Name)
 	}
 
-	command := ""
-	if o.ServerInfo.User == "root" {
-		command = "echo N | "
-	}
+	command := "echo N | "
 	if 0 < len(config.Conf.HTTPProxy) {
 		command += util.ProxyEnv()
 	}
 
+	yumopts := ""
+	if o.getServerInfo().Enablerepo != "" {
+		yumopts = " --enablerepo=" + o.getServerInfo().Enablerepo
+	}
+	if config.Conf.SkipBroken {
+		yumopts += " --skip-broken"
+	}
 	// yum update --changelog doesn't have --color option.
-	command += fmt.Sprintf(" LANG=en_US.UTF-8 yum update --changelog %s", packageNames)
+	command += fmt.Sprintf(" LANGUAGE=en_US.UTF-8 yum %s --changelog update ", yumopts) + packageNames
 
 	r := o.ssh(command, sudo)
 	if !r.isSuccess(0, 1) {
@@ -602,24 +562,24 @@ type distroAdvisoryCveIDs struct {
 }
 
 // Scaning unsecure packages using yum-plugin-security.
-//TODO return whether already expired.
-func (o *redhat) scanUnsecurePackagesUsingYumPluginSecurity() (CvePacksList, error) {
-	if o.Family == "centos" {
+// Amazon, RHEL
+func (o *redhat) scanUnsecurePackagesUsingYumPluginSecurity() (models.VulnInfos, error) {
+	if o.Distro.Family == "centos" {
 		// CentOS has no security channel.
 		// So use yum check-update && parse changelog
-		return CvePacksList{}, fmt.Errorf(
+		return nil, fmt.Errorf(
 			"yum updateinfo is not suppported on CentOS")
 	}
 
 	cmd := "yum --color=never repolist"
-	r := o.ssh(util.PrependProxyEnv(cmd), sudo)
+	r := o.ssh(util.PrependProxyEnv(cmd), o.sudo())
 	if !r.isSuccess() {
 		return nil, fmt.Errorf("Failed to SSH: %s", r)
 	}
 
 	// get advisoryID(RHSA, ALAS) - package name,version
 	cmd = "yum --color=never updateinfo list available --security"
-	r = o.ssh(util.PrependProxyEnv(cmd), sudo)
+	r = o.ssh(util.PrependProxyEnv(cmd), o.sudo())
 	if !r.isSuccess() {
 		return nil, fmt.Errorf("Failed to SSH: %s", r)
 	}
@@ -627,8 +587,8 @@ func (o *redhat) scanUnsecurePackagesUsingYumPluginSecurity() (CvePacksList, err
 
 	// get package name, version, rel to be upgrade.
 	//  cmd = "yum check-update --security"
-	cmd = "LANG=en_US.UTF-8 yum --color=never check-update"
-	r = o.ssh(util.PrependProxyEnv(cmd), sudo)
+	cmd = "LANGUAGE=en_US.UTF-8 yum --color=never check-update"
+	r = o.ssh(util.PrependProxyEnv(cmd), o.sudo())
 	if !r.isSuccess(0, 100) {
 		//returns an exit code of 100 if there are available updates.
 		return nil, fmt.Errorf("Failed to SSH: %s", r)
@@ -639,6 +599,9 @@ func (o *redhat) scanUnsecurePackagesUsingYumPluginSecurity() (CvePacksList, err
 	}
 	o.log.Debugf("%s", pp.Sprintf("%v", updatable))
 
+	// set candidate version info
+	o.Packages.MergeNewVersion(updatable)
+
 	dict := map[string][]models.PackageInfo{}
 	for _, advIDPackNames := range advIDPackNamesList {
 		packInfoList := models.PackageInfoList{}
@@ -656,56 +619,51 @@ func (o *redhat) scanUnsecurePackagesUsingYumPluginSecurity() (CvePacksList, err
 
 	// get advisoryID(RHSA, ALAS) - CVE IDs
 	cmd = "yum --color=never updateinfo --security update"
-	r = o.ssh(util.PrependProxyEnv(cmd), sudo)
+	r = o.ssh(util.PrependProxyEnv(cmd), o.sudo())
 	if !r.isSuccess() {
 		return nil, fmt.Errorf("Failed to SSH: %s", r)
 	}
 	advisoryCveIDsList, err := o.parseYumUpdateinfo(r.Stdout)
 	if err != nil {
-		return CvePacksList{}, err
+		return nil, err
 	}
 	//  pp.Println(advisoryCveIDsList)
 
 	// All information collected.
-	// Convert to CvePacksList.
+	// Convert to VulnInfos.
 	o.log.Info("Fetching CVE details...")
-	result := CvePacksList{}
+	vinfos := models.VulnInfos{}
 	for _, advIDCveIDs := range advisoryCveIDsList {
-		cveDetails, err :=
-			cveapi.CveClient.FetchCveDetails(advIDCveIDs.CveIDs)
-		if err != nil {
-			return nil, err
-		}
-
-		for _, cveDetail := range cveDetails {
+		for _, cveID := range advIDCveIDs.CveIDs {
 			found := false
-			for i, p := range result {
-				if cveDetail.CveID == p.CveID {
+			for i, p := range vinfos {
+				if cveID == p.CveID {
 					advAppended := append(p.DistroAdvisories, advIDCveIDs.DistroAdvisory)
-					result[i].DistroAdvisories = advAppended
+					vinfos[i].DistroAdvisories = advAppended
 
 					packs := dict[advIDCveIDs.DistroAdvisory.AdvisoryID]
-					result[i].Packs = append(result[i].Packs, packs...)
+					vinfos[i].Packages = append(vinfos[i].Packages, packs...)
 					found = true
 					break
 				}
 			}
 
 			if !found {
-				cpinfo := CvePacksInfo{
-					CveID:            cveDetail.CveID,
-					CveDetail:        cveDetail,
+				cpinfo := models.VulnInfo{
+					CveID:            cveID,
 					DistroAdvisories: []models.DistroAdvisory{advIDCveIDs.DistroAdvisory},
-					Packs:            dict[advIDCveIDs.DistroAdvisory.AdvisoryID],
+					Packages:         dict[advIDCveIDs.DistroAdvisory.AdvisoryID],
 				}
-				result = append(result, cpinfo)
+				vinfos = append(vinfos, cpinfo)
 			}
+
 		}
 	}
-	o.log.Info("Done")
-	return result, nil
+	return vinfos, nil
 }
 
+var horizontalRulePattern = regexp.MustCompile(`^=+$`)
+
 func (o *redhat) parseYumUpdateinfo(stdout string) (result []distroAdvisoryCveIDs, err error) {
 	sectionState := Outside
 	lines := strings.Split(stdout, "\n")
@@ -723,7 +681,7 @@ func (o *redhat) parseYumUpdateinfo(stdout string) (result []distroAdvisoryCveID
 		line = strings.TrimSpace(line)
 
 		// find the new section pattern
-		if match, _ := o.isHorizontalRule(line); match {
+		if horizontalRulePattern.MatchString(line) {
 
 			// set previous section's result to return-variable
 			if sectionState == Content {
@@ -750,7 +708,7 @@ func (o *redhat) parseYumUpdateinfo(stdout string) (result []distroAdvisoryCveID
 
 		switch sectionState {
 		case Header:
-			switch o.Family {
+			switch o.Distro.Family {
 			case "centos":
 				// CentOS has no security channel.
 				// So use yum check-update && parse changelog
@@ -817,9 +775,8 @@ func (o *redhat) changeSectionState(state int) (newState int) {
 	return newState
 }
 
-func (o *redhat) isHorizontalRule(line string) (bool, error) {
-	return regexp.MatchString("^=+$", line)
-}
+var rpmPackageArchPattern = regexp.MustCompile(
+	`^[^ ]+\.(i386|i486|i586|i686|k6|athlon|x86_64|noarch|ppc|alpha|sparc)$`)
 
 func (o *redhat) isRpmPackageNameLine(line string) (bool, error) {
 	s := strings.TrimPrefix(line, "ChangeLog for: ")
@@ -829,10 +786,8 @@ func (o *redhat) isRpmPackageNameLine(line string) (bool, error) {
 	}
 	for _, s := range ss {
 		s = strings.TrimRight(s, " \r\n")
-		ok, err := regexp.MatchString(
-			`^[^ ]+\.(i386|i486|i586|i686|k6|athlon|x86_64|noarch|ppc|alpha|sparc)$`, s)
-		if !ok {
-			return false, err
+		if !rpmPackageArchPattern.MatchString(s) {
+			return false, nil
 		}
 	}
 	return true, nil
@@ -855,9 +810,10 @@ func (o *redhat) parseYumUpdateinfoHeaderCentOS(line string) (packs []models.Pac
 	return
 }
 
+var yumHeaderPattern = regexp.MustCompile(`(ALAS-.+): (.+) priority package update for (.+)$`)
+
 func (o *redhat) parseYumUpdateinfoHeaderAmazon(line string) (a models.DistroAdvisory, names []string, err error) {
-	re, _ := regexp.Compile(`(ALAS-.+): (.+) priority package update for (.+)$`)
-	result := re.FindStringSubmatch(line)
+	result := yumHeaderPattern.FindStringSubmatch(line)
 	if len(result) == 4 {
 		a.AdvisoryID = result[1]
 		a.Severity = result[2]
@@ -869,31 +825,36 @@ func (o *redhat) parseYumUpdateinfoHeaderAmazon(line string) (a models.DistroAdv
 	return
 }
 
+var yumCveIDPattern = regexp.MustCompile(`(CVE-\d{4}-\d{4,})`)
+
 func (o *redhat) parseYumUpdateinfoLineToGetCveIDs(line string) []string {
-	re, _ := regexp.Compile(`(CVE-\d{4}-\d{4})`)
-	return re.FindAllString(line, -1)
+	return yumCveIDPattern.FindAllString(line, -1)
 }
 
+var yumAdvisoryIDPattern = regexp.MustCompile(`^ *Update ID : (.*)$`)
+
 func (o *redhat) parseYumUpdateinfoToGetAdvisoryID(line string) (advisoryID string, found bool) {
-	re, _ := regexp.Compile(`^ *Update ID : (.*)$`)
-	result := re.FindStringSubmatch(line)
+	result := yumAdvisoryIDPattern.FindStringSubmatch(line)
 	if len(result) != 2 {
 		return "", false
 	}
 	return strings.TrimSpace(result[1]), true
 }
 
+var yumIssuedPattern = regexp.MustCompile(`^\s*Issued : (\d{4}-\d{2}-\d{2})`)
+
 func (o *redhat) parseYumUpdateinfoLineToGetIssued(line string) (date time.Time, found bool) {
-	return o.parseYumUpdateinfoLineToGetDate(line, `^\s*Issued : (\d{4}-\d{2}-\d{2})`)
+	return o.parseYumUpdateinfoLineToGetDate(line, yumIssuedPattern)
 }
 
+var yumUpdatedPattern = regexp.MustCompile(`^\s*Updated : (\d{4}-\d{2}-\d{2})`)
+
 func (o *redhat) parseYumUpdateinfoLineToGetUpdated(line string) (date time.Time, found bool) {
-	return o.parseYumUpdateinfoLineToGetDate(line, `^\s*Updated : (\d{4}-\d{2}-\d{2})`)
+	return o.parseYumUpdateinfoLineToGetDate(line, yumUpdatedPattern)
 }
 
-func (o *redhat) parseYumUpdateinfoLineToGetDate(line, regexpFormat string) (date time.Time, found bool) {
-	re, _ := regexp.Compile(regexpFormat)
-	result := re.FindStringSubmatch(line)
+func (o *redhat) parseYumUpdateinfoLineToGetDate(line string, regexpPattern *regexp.Regexp) (date time.Time, found bool) {
+	result := regexpPattern.FindStringSubmatch(line)
 	if len(result) != 2 {
 		return date, false
 	}
@@ -904,14 +865,16 @@ func (o *redhat) parseYumUpdateinfoLineToGetDate(line, regexpFormat string) (dat
 	return t, true
 }
 
+var yumDescriptionPattern = regexp.MustCompile(`^\s*Description : `)
+
 func (o *redhat) isDescriptionLine(line string) bool {
-	re, _ := regexp.Compile(`^\s*Description : `)
-	return re.MatchString(line)
+	return yumDescriptionPattern.MatchString(line)
 }
 
+var yumSeverityPattern = regexp.MustCompile(`^ *Severity : (.*)$`)
+
 func (o *redhat) parseYumUpdateinfoToGetSeverity(line string) (severity string, found bool) {
-	re, _ := regexp.Compile(`^ *Severity : (.*)$`)
-	result := re.FindStringSubmatch(line)
+	result := yumSeverityPattern.FindStringSubmatch(line)
 	if len(result) != 2 {
 		return "", false
 	}
@@ -946,7 +909,6 @@ func (o *redhat) extractPackNameVerRel(nameVerRel string) (name, ver, rel string
 
 // parseYumUpdateinfoListAvailable collect AdvisorID(RHSA, ALAS), packages
 func (o *redhat) parseYumUpdateinfoListAvailable(stdout string) (advisoryIDPacksList, error) {
-
 	result := []advisoryIDPacks{}
 	lines := strings.Split(stdout, "\n")
 	for _, line := range lines {
@@ -990,3 +952,12 @@ func (o *redhat) parseYumUpdateinfoListAvailable(stdout string) (advisoryIDPacks
 func (o *redhat) clone() osTypeInterface {
 	return o
 }
+
+func (o *redhat) sudo() bool {
+	switch o.Distro.Family {
+	case "amazon":
+		return false
+	default:
+		return true
+	}
+}

scan/redhat_test.go

@@ -143,11 +143,11 @@ func TestParseYumUpdateinfoLineToGetCveIDs(t *testing.T) {
 			[]string{"CVE-2015-0278"},
 		},
 		{
-			": 1195457 - nodejs-0.10.35 causes undefined symbolsCVE-2015-0278, CVE-2015-0278, CVE-2015-0277",
+			": 1195457 - nodejs-0.10.35 causes undefined symbolsCVE-2015-0278, CVE-2015-0278, CVE-2015-02770000000 ",
 			[]string{
 				"CVE-2015-0278",
 				"CVE-2015-0278",
-				"CVE-2015-0277",
+				"CVE-2015-02770000000",
 			},
 		},
 	}
@@ -451,7 +451,7 @@ Description : The Berkeley Internet Name Domain (BIND) is an implementation of
 	updated, _ := time.Parse("2006-01-02", "2015-09-04")
 
 	r := newRedhat(config.ServerInfo{})
-	r.Family = "redhat"
+	r.Distro = config.Distro{Family: "redhat"}
 
 	var tests = []struct {
 		in  string
@@ -511,7 +511,7 @@ Description : The Berkeley Internet Name Domain (BIND) is an implementation of
 func TestParseYumUpdateinfoAmazon(t *testing.T) {
 
 	r := newRedhat(config.ServerInfo{})
-	r.Family = "amazon"
+	r.Distro = config.Distro{Family: "redhat"}
 
 	issued, _ := time.Parse("2006-01-02", "2015-12-15")
 	updated, _ := time.Parse("2006-01-02", "2015-12-16")
@@ -601,7 +601,7 @@ Description : Package updates are available for Amazon Linux AMI that fix the
 
 func TestParseYumCheckUpdateLines(t *testing.T) {
 	r := newRedhat(config.ServerInfo{})
-	r.Family = "centos"
+	r.Distro = config.Distro{Family: "centos"}
 	stdout := `Loaded plugins: changelog, fastestmirror, keys, protect-packages, protectbase, security
 Loading mirror speeds from cached hostfile
  * base: mirror.fairway.ne.jp
@@ -616,6 +616,7 @@ Obsoleting Packages
 python-libs.i686    2.6.6-64.el6   rhui-REGION-rhel-server-releases
     python-ordereddict.noarch     1.1-3.el6ev    installed
 bind-utils.x86_64                       30:9.3.6-25.P1.el5_11.8          updates
+pytalloc.x86_64                 2.0.7-2.el6                      @CentOS 6.5/6.5
 `
 
 	r.Packages = []models.PackageInfo{
@@ -644,6 +645,11 @@ bind-utils.x86_64                       30:9.3.6-25.P1.el5_11.8          updates
 			Version: "1.0",
 			Release: "1",
 		},
+		{
+			Name:    "pytalloc",
+			Version: "2.0.1",
+			Release: "0",
+		},
 	}
 	var tests = []struct {
 		in  string
@@ -658,6 +664,7 @@ bind-utils.x86_64                       30:9.3.6-25.P1.el5_11.8          updates
 					Release:    "4.el6",
 					NewVersion: "2.3.7",
 					NewRelease: "5.el6",
+					Repository: "base",
 				},
 				{
 					Name:       "bash",
@@ -665,6 +672,7 @@ bind-utils.x86_64                       30:9.3.6-25.P1.el5_11.8          updates
 					Release:    "33",
 					NewVersion: "4.1.2",
 					NewRelease: "33.el6_7.1",
+					Repository: "updates",
 				},
 				{
 					Name:       "python-libs",
@@ -672,6 +680,7 @@ bind-utils.x86_64                       30:9.3.6-25.P1.el5_11.8          updates
 					Release:    "1.1-0",
 					NewVersion: "2.6.6",
 					NewRelease: "64.el6",
+					Repository: "rhui-REGION-rhel-server-releases",
 				},
 				{
 					Name:       "python-ordereddict",
@@ -679,6 +688,7 @@ bind-utils.x86_64                       30:9.3.6-25.P1.el5_11.8          updates
 					Release:    "1",
 					NewVersion: "1.1",
 					NewRelease: "3.el6ev",
+					Repository: "installed",
 				},
 				{
 					Name:       "bind-utils",
@@ -686,6 +696,15 @@ bind-utils.x86_64                       30:9.3.6-25.P1.el5_11.8          updates
 					Release:    "1",
 					NewVersion: "9.3.6",
 					NewRelease: "25.P1.el5_11.8",
+					Repository: "updates",
+				},
+				{
+					Name:       "pytalloc",
+					Version:    "2.0.1",
+					Release:    "0",
+					NewVersion: "2.0.7",
+					NewRelease: "2.el6",
+					Repository: "@CentOS 6.5/6.5",
 				},
 			},
 		},
@@ -709,7 +728,7 @@ bind-utils.x86_64                       30:9.3.6-25.P1.el5_11.8          updates
 
 func TestParseYumCheckUpdateLinesAmazon(t *testing.T) {
 	r := newRedhat(config.ServerInfo{})
-	r.Family = "amazon"
+	r.Distro = config.Distro{Family: "amazon"}
 	stdout := `Loaded plugins: priorities, update-motd, upgrade-helper
 34 package(s) needed for security, out of 71 available
 
@@ -747,6 +766,7 @@ if-not-architecture        100-200                         amzn-main
 					Release:    "0.33.rc1.45.amzn1",
 					NewVersion: "9.8.2",
 					NewRelease: "0.37.rc1.45.amzn1",
+					Repository: "amzn-main",
 				},
 				{
 					Name:       "java-1.7.0-openjdk",
@@ -754,6 +774,7 @@ if-not-architecture        100-200                         amzn-main
 					Release:    "2.6.4.0.0.amzn1",
 					NewVersion: "1.7.0.95",
 					NewRelease: "2.6.4.0.65.amzn1",
+					Repository: "amzn-main",
 				},
 				{
 					Name:       "if-not-architecture",
@@ -761,6 +782,7 @@ if-not-architecture        100-200                         amzn-main
 					Release:    "20",
 					NewVersion: "100",
 					NewRelease: "200",
+					Repository: "amzn-main",
 				},
 			},
 		},
@@ -1110,8 +1132,10 @@ func TestGetChangelogCVELines(t *testing.T) {
 	}
 
 	r := newRedhat(config.ServerInfo{})
-	r.Family = "centos"
-	r.Release = "6.7"
+	r.Distro = config.Distro{
+		Family:  "centos",
+		Release: "6.7",
+	}
 	for _, tt := range testsCentos6 {
 		rpm2changelog, err := r.parseAllChangelog(stdoutCentos6)
 		if err != nil {
@@ -1194,7 +1218,10 @@ func TestGetChangelogCVELines(t *testing.T) {
 		},
 	}
 
-	r.Release = "5.6"
+	r.Distro = config.Distro{
+		Family:  "centos",
+		Release: "5.6",
+	}
 	for _, tt := range testsCentos5 {
 		rpm2changelog, err := r.parseAllChangelog(stdoutCentos5)
 		if err != nil {

scan/serverapi.go

@@ -1,13 +1,35 @@
+/* Vuls - Vulnerability Scanner
+Copyright (C) 2016  Future Architect, Inc. Japan.
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
 package scan
 
 import (
+	"bufio"
 	"fmt"
+	"os"
+	"path/filepath"
+	"strings"
 	"time"
 
 	"github.com/Sirupsen/logrus"
+	"github.com/future-architect/vuls/cache"
 	"github.com/future-architect/vuls/config"
 	"github.com/future-architect/vuls/models"
-	cve "github.com/kotakanbe/go-cve-dictionary/models"
+	"github.com/future-architect/vuls/report"
 )
 
 // Log for localhsot
@@ -20,8 +42,12 @@ type osTypeInterface interface {
 	setServerInfo(config.ServerInfo)
 	getServerInfo() config.ServerInfo
 
-	setDistributionInfo(string, string)
-	getDistributionInfo() string
+	setDistro(string, string)
+	getDistro() config.Distro
+
+	// checkDependencies checks if dependencies are installed on the target server.
+	checkDependencies() error
+	getLackDependencies() []string
 
 	checkIfSudoNoPasswd() error
 	detectPlatform() error
@@ -29,7 +55,6 @@ type osTypeInterface interface {
 
 	checkRequiredPackagesInstalled() error
 	scanPackages() error
-	scanVulnByCpeName() error
 	install() error
 	convertToModel() (models.ScanResult, error)
 
@@ -41,70 +66,21 @@ type osTypeInterface interface {
 	setErrs([]error)
 }
 
-// osPackages included by linux struct
+// osPackages is included by base struct
 type osPackages struct {
 	// installed packages
 	Packages models.PackageInfoList
 
 	// unsecure packages
-	UnsecurePackages CvePacksList
+	VulnInfos models.VulnInfos
 }
 
 func (p *osPackages) setPackages(pi models.PackageInfoList) {
 	p.Packages = pi
 }
 
-func (p *osPackages) setUnsecurePackages(pi []CvePacksInfo) {
-	p.UnsecurePackages = pi
-}
-
-// CvePacksList have CvePacksInfo list, getter/setter, sortable methods.
-type CvePacksList []CvePacksInfo
-
-// CvePacksInfo hold the CVE information.
-type CvePacksInfo struct {
-	CveID            string
-	CveDetail        cve.CveDetail
-	Packs            models.PackageInfoList
-	DistroAdvisories []models.DistroAdvisory // for Aamazon, RHEL, FreeBSD
-	CpeNames         []string
-}
-
-// FindByCveID find by CVEID
-func (s CvePacksList) FindByCveID(cveID string) (pi CvePacksInfo, found bool) {
-	for _, p := range s {
-		if cveID == p.CveID {
-			return p, true
-		}
-	}
-	return CvePacksInfo{CveID: cveID}, false
-}
-
-// immutable
-func (s CvePacksList) set(cveID string, cvePacksInfo CvePacksInfo) CvePacksList {
-	for i, p := range s {
-		if cveID == p.CveID {
-			s[i] = cvePacksInfo
-			return s
-		}
-	}
-	return append(s, cvePacksInfo)
-}
-
-// Len implement Sort Interface
-func (s CvePacksList) Len() int {
-	return len(s)
-}
-
-// Swap implement Sort Interface
-func (s CvePacksList) Swap(i, j int) {
-	s[i], s[j] = s[j], s[i]
-}
-
-// Less implement Sort Interface
-func (s CvePacksList) Less(i, j int) bool {
-	return s[i].CveDetail.CvssScore(config.Conf.Lang) >
-		s[j].CveDetail.CvssScore(config.Conf.Lang)
+func (p *osPackages) setVulnInfos(vi []models.VulnInfo) {
+	p.VulnInfos = vi
 }
 
 func detectOS(c config.ServerInfo) (osType osTypeInterface) {
@@ -151,11 +127,20 @@ func PrintSSHableServerNames() {
 }
 
 // InitServers detect the kind of OS distribution of target servers
-func InitServers(localLogger *logrus.Entry) {
+func InitServers(localLogger *logrus.Entry) error {
 	Log = localLogger
 	servers = detectServerOSes()
+	if len(servers) == 0 {
+		return fmt.Errorf("No scannable servers")
+	}
+
 	containers := detectContainerOSes()
-	servers = append(servers, containers...)
+	if config.Conf.ContainersOnly {
+		servers = containers
+	} else {
+		servers = append(servers, containers...)
+	}
+	return nil
 }
 
 func detectServerOSes() (sshAbleOses []osTypeInterface) {
@@ -188,7 +173,7 @@ func detectServerOSes() (sshAbleOses []osTypeInterface) {
 				Log.Infof("(%d/%d) Detected: %s: %s",
 					i+1, len(config.Conf.Servers),
 					res.getServerInfo().ServerName,
-					res.getDistributionInfo())
+					res.getDistro())
 			}
 		case <-timeout:
 			msg := "Timed out while detecting servers"
@@ -228,7 +213,7 @@ func detectContainerOSes() (actives []osTypeInterface) {
 			defer func() {
 				if p := recover(); p != nil {
 					Log.Debugf("Panic: %s on %s",
-						p, s.getServerInfo().ServerName)
+						p, s.getServerInfo().GetServerName())
 				}
 			}()
 			osTypesChan <- detectContainerOSesOnServer(s)
@@ -246,9 +231,9 @@ func detectContainerOSes() (actives []osTypeInterface) {
 					Log.Errorf("Failed: %s err: %s", sinfo.ServerName, osi.getErrs())
 					continue
 				}
-				oses = append(oses, res...)
+				oses = append(oses, osi)
 				Log.Infof("Detected: %s@%s: %s",
-					sinfo.Container.Name, sinfo.ServerName, osi.getDistributionInfo())
+					sinfo.Container.Name, sinfo.ServerName, osi.getDistro())
 			}
 		case <-timeout:
 			msg := "Timed out while detecting containers"
@@ -350,7 +335,7 @@ func detectContainerOSesOnServer(containerHost osTypeInterface) (oses []osTypeIn
 
 // CheckIfSudoNoPasswd checks whether vuls can sudo with nopassword via SSH
 func CheckIfSudoNoPasswd(localLogger *logrus.Entry) error {
-	timeoutSec := 1 * 15
+	timeoutSec := 15
 	errs := parallelSSHExec(func(o osTypeInterface) error {
 		return o.checkIfSudoNoPasswd()
 	}, timeoutSec)
@@ -397,12 +382,68 @@ func detectPlatforms() []error {
 
 // Prepare installs requred packages to scan vulnerabilities.
 func Prepare() []error {
-	return parallelSSHExec(func(o osTypeInterface) error {
+	errs := parallelSSHExec(func(o osTypeInterface) error {
+		if err := o.checkDependencies(); err != nil {
+			return err
+		}
+		return nil
+	})
+	if len(errs) != 0 {
+		return errs
+	}
+
+	var targets []osTypeInterface
+	for _, s := range servers {
+		deps := s.getLackDependencies()
+		if len(deps) != 0 {
+			targets = append(targets, s)
+		}
+	}
+	if len(targets) == 0 {
+		Log.Info("No need to install dependencies")
+		return nil
+	}
+
+	Log.Info("The following servers need dependencies installed")
+	for _, s := range targets {
+		for _, d := range s.getLackDependencies() {
+			Log.Infof("  - %s on %s", d, s.getServerInfo().GetServerName())
+		}
+	}
+
+	if !config.Conf.AssumeYes {
+		Log.Info("Is this ok to install dependencies on the servers? [y/N]")
+
+		reader := bufio.NewReader(os.Stdin)
+		for {
+			text, err := reader.ReadString('\n')
+			if err != nil {
+				return []error{err}
+			}
+			switch strings.TrimSpace(text) {
+			case "", "N", "n":
+				return nil
+			case "y", "Y":
+				goto yes
+			default:
+				Log.Info("Please enter y or N")
+			}
+		}
+	}
+
+yes:
+	servers = targets
+	errs = parallelSSHExec(func(o osTypeInterface) error {
 		if err := o.install(); err != nil {
 			return err
 		}
 		return nil
 	})
+	if len(errs) != 0 {
+		return errs
+	}
+	Log.Info("All dependencies were installed correctly")
+	return nil
 }
 
 // Scan scan
@@ -417,14 +458,42 @@ func Scan() []error {
 		return errs
 	}
 
+	if err := setupCangelogCache(); err != nil {
+		return []error{err}
+	}
+
+	defer func() {
+		if cache.DB != nil {
+			cache.DB.Close()
+		}
+	}()
+
 	Log.Info("Scanning vulnerable OS packages...")
-	if errs := scanPackages(); errs != nil {
+	scannedAt := time.Now()
+	dir, err := ensureResultDir(scannedAt)
+	if err != nil {
+		return []error{err}
+	}
+	if errs := scanVulns(dir, scannedAt); errs != nil {
 		return errs
 	}
 
-	Log.Info("Scanning vulnerable software specified in the CPE...")
-	if errs := scanVulnByCpeName(); errs != nil {
-		return errs
+	return nil
+}
+
+func setupCangelogCache() error {
+	needToSetupCache := false
+	for _, s := range servers {
+		switch s.getDistro().Family {
+		case "ubuntu", "debian":
+			needToSetupCache = true
+			break
+		}
+	}
+	if needToSetupCache {
+		if err := cache.SetupBolt(config.Conf.CacheDBPath, Log); err != nil {
+			return err
+		}
 	}
 	return nil
 }
@@ -436,31 +505,67 @@ func checkRequiredPackagesInstalled() []error {
 	}, timeoutSec)
 }
 
-func scanPackages() []error {
+func scanVulns(jsonDir string, scannedAt time.Time) []error {
+	var results models.ScanResults
 	timeoutSec := 120 * 60
-	return parallelSSHExec(func(o osTypeInterface) error {
-		return o.scanPackages()
-	}, timeoutSec)
-
-}
-
-// scanVulnByCpeName search vulnerabilities that specified in config file.
-func scanVulnByCpeName() []error {
-	timeoutSec := 30 * 60
-	return parallelSSHExec(func(o osTypeInterface) error {
-		return o.scanVulnByCpeName()
-	}, timeoutSec)
-
-}
-
-// GetScanResults returns Scan Resutls
-func GetScanResults() (results models.ScanResults, err error) {
-	for _, s := range servers {
-		r, err := s.convertToModel()
-		if err != nil {
-			return results, fmt.Errorf("Failed converting to model: %s", err)
+	errs := parallelSSHExec(func(o osTypeInterface) error {
+		if err := o.scanPackages(); err != nil {
+			return err
 		}
+
+		r, err := o.convertToModel()
+		if err != nil {
+			return err
+		}
+		r.ScannedAt = scannedAt
 		results = append(results, r)
+
+		return nil
+	}, timeoutSec)
+
+	config.Conf.FormatJSON = true
+	ws := []report.ResultWriter{
+		report.LocalFileWriter{CurrentDir: jsonDir},
 	}
-	return
+	for _, w := range ws {
+		if err := w.Write(results...); err != nil {
+			return []error{
+				fmt.Errorf("Failed to write summary report: %s", err),
+			}
+		}
+	}
+	if errs != nil {
+		return errs
+	}
+
+	report.StdoutWriter{}.WriteScanSummary(results...)
+	return nil
+}
+
+func ensureResultDir(scannedAt time.Time) (currentDir string, err error) {
+	jsonDirName := scannedAt.Format(time.RFC3339)
+
+	resultsDir := config.Conf.ResultsDir
+	if len(resultsDir) == 0 {
+		wd, _ := os.Getwd()
+		resultsDir = filepath.Join(wd, "results")
+	}
+	jsonDir := filepath.Join(resultsDir, jsonDirName)
+	if err := os.MkdirAll(jsonDir, 0700); err != nil {
+		return "", fmt.Errorf("Failed to create dir: %s", err)
+	}
+
+	symlinkPath := filepath.Join(resultsDir, "current")
+	if _, err := os.Lstat(symlinkPath); err == nil {
+		if err := os.Remove(symlinkPath); err != nil {
+			return "", fmt.Errorf(
+				"Failed to remove symlink. path: %s, err: %s", symlinkPath, err)
+		}
+	}
+
+	if err := os.Symlink(jsonDir, symlinkPath); err != nil {
+		return "", fmt.Errorf(
+			"Failed to create symlink: path: %s, err: %s", symlinkPath, err)
+	}
+	return jsonDir, nil
 }

scan/serverapi_test.go

@@ -1,47 +1 @@
 package scan
-
-import "testing"
-
-func TestPackageCveInfosSetGet(t *testing.T) {
-	var test = struct {
-		in  []string
-		out []string
-	}{
-		[]string{
-			"CVE1",
-			"CVE2",
-			"CVE3",
-			"CVE1",
-			"CVE1",
-			"CVE2",
-			"CVE3",
-		},
-		[]string{
-			"CVE1",
-			"CVE2",
-			"CVE3",
-		},
-	}
-
-	//  var ps packageCveInfos
-	var ps CvePacksList
-	for _, cid := range test.in {
-		ps = ps.set(cid, CvePacksInfo{CveID: cid})
-	}
-
-	if len(test.out) != len(ps) {
-		t.Errorf("length: expected %d, actual %d", len(test.out), len(ps))
-	}
-
-	for i, expectedCid := range test.out {
-		if expectedCid != ps[i].CveID {
-			t.Errorf("expected %s, actual %s", expectedCid, ps[i].CveID)
-		}
-	}
-	for _, cid := range test.in {
-		p, _ := ps.FindByCveID(cid)
-		if p.CveID != cid {
-			t.Errorf("expected %s, actual %s", cid, p.CveID)
-		}
-	}
-}

scan/sshutil.go

@@ -26,7 +26,6 @@ import (
 	"net"
 	"os"
 	"os/exec"
-	"runtime"
 	"strings"
 	"syscall"
 	"time"
@@ -89,7 +88,7 @@ func parallelSSHExec(fn func(osTypeInterface) error, timeoutSec ...int) (errs []
 			defer func() {
 				if p := recover(); p != nil {
 					logrus.Debugf("Panic: %s on %s",
-						p, s.getServerInfo().ServerName)
+						p, s.getServerInfo().GetServerName())
 				}
 			}()
 			if err := fn(s); err != nil {
@@ -100,7 +99,7 @@ func parallelSSHExec(fn func(osTypeInterface) error, timeoutSec ...int) (errs []
 					err,
 				)
 			} else {
-				resChan <- s.getServerInfo().ServerName
+				resChan <- s.getServerInfo().GetServerName()
 			}
 		}(s)
 	}
@@ -129,7 +128,7 @@ func parallelSSHExec(fn func(osTypeInterface) error, timeoutSec ...int) (errs []
 	var timedoutSnames []string
 	if isTimedout {
 		for _, s := range servers {
-			name := s.getServerInfo().ServerName
+			name := s.getServerInfo().GetServerName()
 			found := false
 			for _, t := range snames {
 				if name == t {
@@ -150,10 +149,10 @@ func parallelSSHExec(fn func(osTypeInterface) error, timeoutSec ...int) (errs []
 }
 
 func sshExec(c conf.ServerInfo, cmd string, sudo bool, log ...*logrus.Entry) (result sshResult) {
-	if isSSHExecNative() {
-		result = sshExecNative(c, cmd, sudo)
-	} else {
+	if conf.Conf.SSHExternal {
 		result = sshExecExternal(c, cmd, sudo)
+	} else {
+		result = sshExecNative(c, cmd, sudo)
 	}
 
 	logger := getSSHLogger(log...)
@@ -161,10 +160,6 @@ func sshExec(c conf.ServerInfo, cmd string, sudo bool, log ...*logrus.Entry) (re
 	return
 }
 
-func isSSHExecNative() bool {
-	return runtime.GOOS == "windows" || !conf.Conf.SSHExternal
-}
-
 func sshExecNative(c conf.ServerInfo, cmd string, sudo bool) (result sshResult) {
 	result.Servername = c.ServerName
 	result.Host = c.Host
@@ -299,9 +294,10 @@ func getSSHLogger(log ...*logrus.Entry) *logrus.Entry {
 func decolateCmd(c conf.ServerInfo, cmd string, sudo bool) string {
 	if sudo && c.User != "root" && !c.IsContainer() {
 		cmd = fmt.Sprintf("sudo -S %s", cmd)
+		cmd = strings.Replace(cmd, "|", "| sudo ", -1)
 	}
 
-	if c.Family != "FreeBSD" {
+	if c.Distro.Family != "FreeBSD" {
 		// set pipefail option. Bash only
 		// http://unix.stackexchange.com/questions/14270/get-exit-status-of-process-thats-piped-to-another
 		cmd = fmt.Sprintf("set -o pipefail; %s", cmd)
@@ -318,7 +314,7 @@ func decolateCmd(c conf.ServerInfo, cmd string, sudo bool) string {
 }
 
 func getAgentAuth() (auth ssh.AuthMethod, ok bool) {
-	if sock := os.Getenv("SSH_AUTH_SOCK"); len(sock) > 0 {
+	if sock := os.Getenv("SSH_AUTH_SOCK"); 0 < len(sock) {
 		if agconn, err := net.Dial("unix", sock); err == nil {
 			ag := agent.NewClient(agconn)
 			auth = ssh.PublicKeysCallback(ag.Signers)

setup/docker/README.ja.md

@@ -1,101 +0,0 @@
-# Vuls on Docker
-
-## What's Vuls-On-Docker
-
-- 数個のコマンドを実行するだけでVulsとvulsrepoのセットアップが出来るスクリプト
-- Dockerコンテナ上にVulsと[vulsrepo](https://github.com/usiusi360/vulsrepo)をセットアップ可能
-- スキャン結果をvulsrepoでブラウザで分析可能
-- 脆弱性データベースの更新が可能
-- モジュールのアップデートが可能
-
-## Setting up your machine
-	
-1. [Install Docker](https://docs.docker.com/engine/installation/)
-2. [Install Docker-Compose](https://docs.docker.com/compose/install/)
-3. 実行前に以下のコマンドが実行可能なことを確認する
-
-	```
-	$ docker version
-	$ docker-compose version
-	```
-
-4. Vulsをgit clone
-	```
-	mkdir work
-	cd work
-	git clone https://github.com/future-architect/vuls.git
-	cd vuls/setup/docker
-	```
-
-## Start A Vuls Container
-
-- 以下のコマンドを実行してコンテナをビルドする
-
-	```
-	$ docker-compose up -d
-	```
-
-## Setting up Vuls
-
-1. スキャン対象サーバのSSH秘密鍵を保存(vuls/setup/docker/conf/)する
-2. config.toml(vuls/docker/conf/config.toml) を環境に合わせて作成する
-	
-	```
-	[servers]
-
-  	[servers.172-31-4-82]
-  	host        = "172.31.4.82"
-  	user        = "ec2-user"
-  	keyPath     = "conf/id_rsa"
-	```
-
-## Fetch Vulnerability database
-
-- NVDから脆弱性データベースを取得する
-	```
-	$ docker exec -t vuls scripts/fetch_nvd_all.sh
-	```
-
-- レポートを日本語化する場合は、JVNから脆弱性データを取得する
-	```
-	$ docker exec -t vuls scripts/fetch_jvn_all.sh
-	```
-
-## Scan servers with Vuls-On-Docker
-
-- スキャンを実行する
-
-	```
-	$ docker exec -t vuls vuls prepare -config=conf/config.toml
-	$ docker exec -t vuls scripts/scan_for_vulsrepo.sh
-	```
-
-## See the results in a browser 
-
-```
-http://${Vuls_Host}/vulsrepo/
-```
-
-# Update modules
-
-- vuls, go-cve-dictionary, vulsrepoのモジュールをアップデートする
-	```
-	$ docker exec -t vuls scripts/update_modules.sh
-	```
-
-# Update Vulnerability database
-
-- NVDの過去２年分の脆弱性データベースを更新する
-	```
-	$ docker exec -t vuls scripts/fetch_nvd_last2y.sh
-	```
-
-- JVNの過去１ヶ月分の脆弱性データベースを更新する
-	```
-	$ docker exec -t vuls scripts/fetch_jvn_month.sh
-	```
-
-- JVNの過去1週間分の脆弱性データベースを更新する
-	```
-	$ docker exec -t vuls scripts/fetch_jvn_week.sh
-	```

setup/docker/README.md

@@ -1,87 +1,195 @@
-# Vuls on Docker
+# Vuls Docker components
 
-## What's Vuls-On-Docker
+This is the Git repo of the official Docker image for vuls.
 
-- This is a dockernized-Vuls with vulsrepo UI in it.
-- It's designed to reduce the cost of installation and the dependencies that vuls requires.
-- You can run install and run Vuls on your machine with only a few commands.
-- The result can be viewed with a browser
+# Supported tags and respective `Dockerfile` links
 
-## Setting up your machine
-	
-1. [Install Docker](https://docs.docker.com/engine/installation/)
-2. [Install Docker-Compose](https://docs.docker.com/compose/install/)
-3. Make sure that you can run the following commands before you move on.
+- go-cve-dictionary
+  - [`latest` (*go-cve-dictionary:latest Dockerfile*)]()
+- vuls
+  - [`latest` (*vuls:latest Dockerfile*)]()
+- vulsrepo
+  - [`latest` (*vulsrepo:latest Dockerfile*)]()
 
-	```
-	$ docker version
-	$ docker-compose version
-	```
-	
-4. git clone vuls
-	```
-	mkdir work
-	cd work
-	git clone https://github.com/future-architect/vuls.git
-	cd vuls/setup/docker
-	```
+This image version is same as the github repository version.
 
+# Caution
 
+This image is built per commit.
+If you want to use the latest docker image, you should remove the existing image, and pull it once again.
 
-## Start A Vuls Container
+1. Confirm your vuls version
 
-- Execute the following command to build and run a Vuls Container
+- go-cve-dictionary
 
-	```
-	$ docker-compose up -d
-	```
+```console
+$ docker run  --rm  vuls/go-cve-dictionary -v
 
-## Setting up Vuls
-
-1. Locate ssh-keys of targer servers in (vuls/docker/conf/)
-2. Create and ajust config.toml(vuls/docker/conf/config.toml) to your environment
-	
-	```
-	[servers]
-
-  	[servers.172-31-4-82]
-  	host        = "172.31.4.82"
-  	user        = "ec2-user"
-  	keyPath     = "conf/id_rsa"
-	```
-
-## Fetch Vulnerability database
-
-- Fetch Vulnerability database from NVD
-	```
-	$ docker exec -t vuls scripts/fetch_nvd_all.sh
-	```
-
-## Scan servers with Vuls-On-Docker
-
-- Use the embedded script to scan servers for vulsrepo(or run whatever with docker exec)
-
-	```
-	$ docker exec -t vuls vuls prepare -config=conf/config.toml
-	$ docker exec -t vuls scripts/scan_for_vulsrepo.sh
-	```
-
-## See the results in a browser 
-
-```
-http://${Vuls_Host}/vulsrepo/
+go-cve-dictionary v0.0.xxx xxxx
 ```
 
-# Update modules
+- vuls
 
-- update vuls, go-cve-dictionary, vulsrepo
-	```
-	$ docker exec -t vuls scripts/update_modules.sh
-	```
+```console
+$ docker run  --rm  vuls/vuls -v
 
-# Update Vulnerability database
+vuls v0.0.xxx xxxx
+```
 
-- Fetch Vulnerability database from NVD
-	```
-	$ docker exec -t vuls scripts/fetch_nvd_last2y.sh
-	```
+2. Remove your old docker images
+
+- go-cve-dictionary
+
+```
+$ docker rmi vuls/go-cve-dictionary
+```
+
+```
+$ docker rmi vuls/vuls
+```
+
+- vuls
+
+```
+$ docker rmi vuls/vuls
+```
+
+3. Pull new vuls docker images
+
+- go-cve-dictionary
+
+```
+$ docker pull vuls/go-cve-dictionary
+```
+
+- vuls
+
+```
+$ docker pull vuls/vuls
+```
+
+4. Confirm your vuls version
+
+```console
+$ docker run  --rm  vuls/go-cve-dictionary -v
+
+go-cve-dictionary v0.1.xxx xxxx
+```
+
+- vuls
+
+```console
+$ docker run  --rm  vuls/vuls -v
+
+vuls v0.1.xxx xxxx
+```
+
+
+# How to use this image
+
+1. fetch nvd (vuls/go-cve-dictionary)
+1. configuration (vuls/vuls)
+1. prepare (vuls/vuls)
+1. scan (vuls/vuls)
+1. vulsrepo (vuls/vulsrepo)
+
+## Step1. Fetch NVD
+
+```console
+$ for i in {2002..2016}; do \
+    docker run --rm -it \
+    -v $PWD:/vuls \
+    -v $PWD/go-cve-dictionary-log:/var/log/vuls \
+    vuls/go-cve-dictionary fetchnvd -years $i; \
+  done
+```
+
+## Step2. Configuration
+
+Create config.toml referring to [this](https://github.com/future-architect/vuls#configuration).
+
+```toml
+[servers]
+
+[servers.amazon]
+host         = "54.249.93.16"
+port        = "22"
+user        = "vuls-user"
+keyPath     = "/root/.ssh/id_rsa" # path to ssh private key in docker
+```
+　
+
+```console
+$ docker run --rm \
+    -v ~/.ssh:/root/.ssh:ro \
+    -v $PWD:/vuls \
+    -v $PWD/vuls-log:/var/log/vuls \
+    vuls/vuls configtest \
+    -config=./config.toml # path to config.toml in docker
+```
+
+## Step3. Prepare
+
+```console
+$ docker run --rm \
+    -v ~/.ssh:/root/.ssh:ro \
+    -v $PWD:/vuls \
+    -v $PWD/vuls-log:/var/log/vuls \
+    vuls/vuls prepare \
+    -config=./config.toml # path to config.toml in docker
+```
+
+## Step4. Scan
+
+```console
+$ docker run --rm -it \
+    -v ~/.ssh:/root/.ssh:ro \
+    -v $PWD:/vuls \
+    -v $PWD/vuls-log:/var/log/vuls \
+    -v /etc/localtime:/etc/localtime:ro \
+    -e "TZ=Asia/Tokyo" \
+    vuls/vuls scan \
+    -config=./config.toml # path to config.toml in docker
+```
+
+## Step5. Report
+
+```console
+$ docker run --rm -it \
+    -v ~/.ssh:/root/.ssh:ro \
+    -v $PWD:/vuls \
+    -v $PWD/vuls-log:/var/log/vuls \
+    -v /etc/localtime:/etc/localtime:ro \
+    vuls/vuls report \
+    -cvedb-path=/vuls/cve.sqlite3 \
+    -format-short-text \
+    -config=./config.toml # path to config.toml in docker
+```
+
+## Step6. vulsrepo
+
+```console
+$docker run -dt \
+    -v $PWD:/vuls \
+    -p 80:80 \
+    vuls/vulsrepo
+```
+
+# User Feedback
+
+## Documentation
+
+Documentation for this image is stored in the [`docker/` directory]() of the [`future-architect/vuls` GitHub repo](https://github.com/future-architect/vuls). 
+
+## Issues
+
+If you have any problems with or questions about this image, please contact us through a [GitHub issue](https://github.com/future-architect/vuls/issues). 
+
+## Contributing
+
+1. fork a repository: github.com/future-architect/vuls to github.com/you/repo
+1. get original code: go get github.com/future-architect/vuls
+1. work on original code
+1. add remote to your repo: git remote add myfork https://github.com/you/repo.git
+1. push your changes: git push myfork
+1. create a new Pull Request

setup/docker/docker-compose.yml

@@ -1,11 +0,0 @@
-version: '2'
-services:
-  vuls:
-    container_name: vuls
-    build: ./dockerfile
-    image: vuls-docker:0.1
-    volumes:
-    - ./conf:/opt/vuls/conf
-    ports:
-      - "80:80"
-

setup/docker/dockerfile/Dockerfile

@@ -1,89 +0,0 @@
-FROM buildpack-deps:jessie-scm
-
-# golang Install
-RUN apt-get update && apt-get install -y --no-install-recommends \
-        g++ \
-        gcc \
-        libc6-dev \
-        make \
-        curl \
-    && rm -rf /var/lib/apt/lists/*
-
-ENV GOLANG_VERSION 1.6.2
-ENV GOLANG_DOWNLOAD_URL https://golang.org/dl/go$GOLANG_VERSION.linux-amd64.tar.gz
-ENV GOLANG_DOWNLOAD_SHA256 e40c36ae71756198478624ed1bb4ce17597b3c19d243f3f0899bb5740d56212a
-
-RUN curl -fsSL "$GOLANG_DOWNLOAD_URL" -o golang.tar.gz \
-    && echo "$GOLANG_DOWNLOAD_SHA256  golang.tar.gz" | sha256sum -c - \
-    && tar -C /usr/local -xzf golang.tar.gz \
-    && rm golang.tar.gz
-
-ENV GOPATH /go
-ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH
-
-RUN mkdir -p "$GOPATH/src" "$GOPATH/bin" && chmod -R 777 "$GOPATH"
-
-# glide install
-ENV GLIDE_VERSION 0.10.2
-ENV GLIDE_DOWNLOAD_URL https://github.com/Masterminds/glide/releases/download/$GLIDE_VERSION/glide-$GLIDE_VERSION-linux-amd64.tar.gz
-RUN curl -fsSL "$GLIDE_DOWNLOAD_URL" -o glide.tar.gz \
-    && mkdir /usr/local/glide \
-    && tar -C /usr/local/glide -xzf glide.tar.gz \
-    && ln -s /usr/local/glide/linux-amd64/glide /usr/local/bin/ \
-    && rm glide.tar.gz
-
-# nginx Install
-RUN apt-key adv --keyserver hkp://pgp.mit.edu:80 --recv-keys 573BFD6B3D8FBC641079A6ABABF5BD827BD9BF62 \
-    && echo "deb http://nginx.org/packages/mainline/debian/ jessie nginx" >> /etc/apt/sources.list \
-    && apt-get update \
-    && apt-get install --no-install-recommends --no-install-suggests -y \
-                        ca-certificates \
-                        nginx \
-                        nginx-module-xslt \
-                        nginx-module-geoip \
-                        nginx-module-image-filter \
-                        nginx-module-perl \
-                        nginx-module-njs \
-                        gettext-base \
-                        wget \
-                        unzip \
-    && rm -rf /var/lib/apt/lists/*
-
-RUN ln -sf /dev/stdout /var/log/nginx/access.log \
-        && ln -sf /dev/stderr /var/log/nginx/error.log
-
-COPY nginx.conf /etc/nginx/nginx.conf
-
-#Vuls Install
-ENV VULS_ROOT /opt/vuls
-RUN mkdir -p /var/log/vuls ${VULS_ROOT}/conf /root/.ssh/
-RUN chmod 700 -R /var/log/vuls $VULS_ROOT
-#  RUN go get github.com/kotakanbe/go-cve-dictionary
-#  RUN go get github.com/future-architect/vuls
-
-RUN go get -v -d github.com/kotakanbe/go-cve-dictionary \
-    && cd $GOPATH/src/github.com/kotakanbe/go-cve-dictionary \
-    && glide install \
-    && go install
-
-RUN go get -v -d github.com/future-architect/vuls \
-    && cd $GOPATH/src/github.com/future-architect/vuls \
-    && glide install \
-    && go install
-
-# Copy custom Scripts
-COPY ./scripts/ ${VULS_ROOT}/scripts
-RUN chmod 755 ${VULS_ROOT}/scripts/*
-
-
-#Vulrepo Install
-RUN git clone https://github.com/usiusi360/vulsrepo /tmp/vulsrepo
-RUN mkdir /usr/share/nginx/html/vulsrepo/
-RUN cp -rp /tmp/vulsrepo/src/* /usr/share/nginx/html/vulsrepo
-RUN rm -rf /tmp/vulsrepo
-
-#Home
-WORKDIR /opt/vuls
-EXPOSE 80 443
-CMD ["nginx", "-g", "daemon off;"]
-

setup/docker/dockerfile/nginx.conf

@@ -1,32 +0,0 @@
-
-user  root;
-worker_processes  1;
-
-error_log  /var/log/nginx/error.log warn;
-pid        /var/run/nginx.pid;
-
-
-events {
-    worker_connections  1024;
-}
-
-
-http {
-    include       /etc/nginx/mime.types;
-    default_type  application/octet-stream;
-
-    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
-                      '$status $body_bytes_sent "$http_referer" '
-                      '"$http_user_agent" "$http_x_forwarded_for"';
-
-    access_log  /var/log/nginx/access.log  main;
-
-    sendfile        on;
-    #tcp_nopush     on;
-
-    keepalive_timeout  65;
-
-    #gzip  on;
-
-    include /etc/nginx/conf.d/*.conf;
-}

setup/docker/dockerfile/scripts/fetch_jvn_all.sh

@@ -1,6 +0,0 @@
-#!/bin/bash
-VULS_ROOT=/opt/vuls
-#VULS_CONF=${VULS_ROOT}/conf
-cd $VULS_ROOT
-go-cve-dictionary fetchjvn -entire
-

setup/docker/dockerfile/scripts/fetch_jvn_month.sh

@@ -1,6 +0,0 @@
-#!/bin/bash
-VULS_ROOT=/opt/vuls
-#VULS_CONF=${VULS_ROOT}/conf
-cd $VULS_ROOT
-go-cve-dictionary fetchjvn -month
-

setup/docker/dockerfile/scripts/fetch_jvn_week.sh

@@ -1,6 +0,0 @@
-#!/bin/bash
-VULS_ROOT=/opt/vuls
-#VULS_CONF=${VULS_ROOT}/conf
-cd $VULS_ROOT
-go-cve-dictionary fetchjvn -week
-

setup/docker/dockerfile/scripts/fetch_nvd_all.sh

@@ -1,6 +0,0 @@
-#!/bin/bash
-VULS_ROOT=/opt/vuls
-#VULS_CONF=${VULS_ROOT}/conf
-cd $VULS_ROOT
-for i in {2002..2016}; do go-cve-dictionary fetchnvd -years $i; done
-

setup/docker/dockerfile/scripts/fetch_nvd_last2y.sh

@@ -1,6 +0,0 @@
-#!/bin/bash
-VULS_ROOT=/opt/vuls
-#VULS_CONF=${VULS_ROOT}/conf
-cd $VULS_ROOT
-go-cve-dictionary fetchnvd -last2y
-

setup/docker/dockerfile/scripts/scan_for_vulsrepo.sh

@@ -1,7 +0,0 @@
-#!/bin/bash
-VULS_ROOT=/opt/vuls
-VULS_CONF=${VULS_ROOT}/conf
-NGINX_VULSREPO_ROOT=/usr/share/nginx/html/vulsrepo
-cd $VULS_ROOT
-vuls scan -report-json --cve-dictionary-dbpath=${VULS_ROOT}/cve.sqlite3 -config=${VULS_CONF}/config.toml 
-ln -sf ${VULS_ROOT}/results/current ${NGINX_VULSREPO_ROOT}/current

setup/docker/dockerfile/scripts/update_modules.sh

@@ -1,17 +0,0 @@
-#!/bin/bash
-
-cd $GOPATH/src/github.com/future-architect/vuls
-git pull origin master
-glide install
-go install
-
-
-cd $GOPATH/src/github.com/kotakanbe/go-cve-dictionary
-git pull origin master
-glide install
-go install
-
-git clone https://github.com/usiusi360/vulsrepo /tmp/vulsrepo
-cp -rp /tmp/vulsrepo/src/* /usr/share/nginx/html/vulsrepo
-rm -rf /tmp/vulsrepo
-

setup/docker/go-cve-dictionary/latest/Dockerfile

@@ -0,0 +1,19 @@
+FROM golang:latest
+
+MAINTAINER hikachan sadayuki-matsuno
+
+ENV REPOSITORY github.com/kotakanbe/go-cve-dictionary
+ENV LOGDIR /var/log/vuls
+ENV WORKDIR /vuls
+# go-cve-dictionary install
+RUN git clone https://$REPOSITORY.git $GOPATH/src/$REPOSITORY \
+    && cd $GOPATH/src/$REPOSITORY \
+    && make install \
+    && mkdir -p $LOGDIR
+
+VOLUME [$WORKDIR, $LOGDIR]
+WORKDIR $WORKDIR
+ENV PWD $WORKDIR
+
+ENTRYPOINT ["go-cve-dictionary"]
+CMD ["--help"]

setup/docker/go-cve-dictionary/latest/README.md

@@ -0,0 +1,89 @@
+# go-cve-dictionary-Docker
+
+This is the Git repo of the official Docker image for go-cve-dictionary.
+See the [Hub page](https://hub.docker.com/r/vuls/go-cve-dictionary/) for the full readme on how to use the Docker image and for information regarding contributing and issues.
+
+# Supported tags and respective `Dockerfile` links
+
+- [`latest` (*go-cve-dictionary:latest Dockerfile*)](https://github.com/future-architect/vuls/blob/master/setup/docker/go-cve-dictionary/latest/Dockerfile)
+
+# Caution
+
+This image is built per commit.
+If you want to use the latest docker image, you should remove the existing image, and pull it once again.
+
+- Remove old docker image
+
+```
+$ docker rmi vuls/go-cve-dictionary
+```
+
+- Pull new docker image
+
+```
+$ docker pull vuls/go-cve-dictionary
+```
+
+# What is go-cve-dictionary?
+
+This is tool to build a local copy of the NVD (National Vulnerabilities Database) [1] and the Japanese JVN [2], which contain security vulnerabilities according to their CVE identifiers [3] including exhaustive information and a risk score. The local copy is generated in sqlite format, and the tool has a server mode for easy querying.
+
+[1] https://en.wikipedia.org/wiki/National_Vulnerability_Database  
+[2] https://en.wikipedia.org/wiki/Common_Vulnerabilities_and_Exposures  
+[3] http://jvndb.jvn.jp/apis/termsofuse.html  
+
+# How to use this image
+
+## check vuls version
+
+```
+$ docker run --rm vuls/go-cve-dictionary -v
+```
+
+## fetchnvd
+
+```console
+$ for i in {2002..2016}; do \
+    docker run --rm -it \
+    -v $PWD:/vuls \
+    -v $PWD/go-cve-dictionary-log:/var/log/vuls \
+    vuls/go-cve-dictionary fetchnvd -years $i; \
+  done
+```
+
+## server
+
+```console
+$ docker run -dt \
+    --name go-cve-dictionary \
+    -v $PWD:/vuls \
+    -v $PWD/go-cve-dictionary-log:/var/log/vuls \
+    --expose 1323 \
+    -p 1323:1323 \
+    vuls/go-cve-dictionary server --bind=0.0.0.0
+```
+
+Prease refer to [this](https://hub.docker.com/r/vuls/go-cve-dictionary).
+
+## vuls
+
+Please refer to [this](https://hub.docker.com/r/vuls/vuls/).
+
+# User Feedback
+
+## Documentation
+
+Documentation for this image is stored in the [`docker/` directory](https://github.com/future-architect/vuls/tree/master/setup/docker) of the [`future-architect/vuls` GitHub repo](https://github.com/future-architect/vuls). 
+
+## Issues
+
+If you have any problems with or questions about this image, please contact us through a [GitHub issue](https://github.com/future-architect/vuls/issues). 
+
+## Contributing
+
+1. fork a repository: github.com/future-architect/vuls to github.com/you/repo
+1. get original code: go get github.com/future-architect/vuls
+1. work on original code
+1. add remote to your repo: git remote add myfork https://github.com/you/repo.git
+1. push your changes: git push myfork
+1. create a new Pull Request

setup/docker/vuls/latest/Dockerfile

@@ -0,0 +1,19 @@
+FROM golang:latest
+
+MAINTAINER hikachan sadayuki-matsuno
+
+ENV REPOSITORY github.com/future-architect/vuls
+ENV LOGDIR /var/log/vuls
+ENV WORKDIR /vuls
+# go-cve-dictionary install
+RUN git clone https://$REPOSITORY.git $GOPATH/src/$REPOSITORY \
+    && cd $GOPATH/src/$REPOSITORY \
+    && make install \
+    && mkdir -p $LOGDIR
+
+VOLUME [$WORKDIR, $LOGDIR]
+WORKDIR $WORKDIR
+ENV PWD $WORKDIR
+
+ENTRYPOINT ["vuls"]
+CMD ["--help"]

setup/docker/vuls/latest/README.md

@@ -0,0 +1,134 @@
+# Vuls-Docker
+
+This is the Git repo of the official Docker image for vuls.
+See the [Hub page](https://hub.docker.com/r/vuls/vuls/) for the full readme on how to use the Docker image and for information regarding contributing and issues.
+
+# Supported tags and respective `Dockerfile` links
+
+- [`latest` (*vuls:latest Dockerfile*)](https://github.com/future-architect/vuls/blob/master/setup/docker/vuls/latest/Dockerfile)
+
+# Caution
+
+This image is built per commit.
+If you want to use the latest docker image, you should remove the existing image, and pull it once again.
+
+- Remove old docker image
+
+```
+$ docker rmi vuls/vuls
+```
+
+- Pull new docker image
+
+```
+$ docker pull vuls/vuls
+```
+
+# What is Vuls?
+
+Vuls is the Vulnerability scanner for Linux/FreeBSD, agentless, written in golang.
+Please see the [Documentation](https://github.com/future-architect/vuls)
+
+![logo](https://github.com/future-architect/vuls/blob/master/img/vuls_logo.png?raw=true)
+
+# How to use this image
+
+## check vuls version
+
+```
+$ docker run  --rm  vuls/vuls -v
+```
+
+## configtest
+
+Create config.toml referring to [this](https://github.com/future-architect/vuls#configuration).
+
+```toml
+[servers]
+
+[servers.amazon]
+host         = "54.249.93.16"
+port        = "22"
+user        = "vuls-user"
+keyPath     = "/root/.ssh/id_rsa"  # path to ssh private key in docker
+```
+　
+
+```console
+$ docker run --rm \
+    -v ~/.ssh:/root/.ssh:ro \
+    -v $PWD:/vuls \
+    -v $PWD/vuls-log:/var/log/vuls \
+    vuls/vuls configtest
+```
+
+
+## prepare
+
+```console
+$ docker run --rm \
+    -v ~/.ssh:/root/.ssh:ro \
+    -v $PWD:/vuls \
+    -v $PWD/vuls-log:/var/log/vuls \
+    vuls/vuls prepare \
+    -config=./config.toml # path to config.toml in docker
+```
+
+## scan
+
+```console
+$ docker run --rm -it \
+    -v ~/.ssh:/root/.ssh:ro \
+    -v $PWD:/vuls \
+    -v $PWD/vuls-log:/var/log/vuls \
+    -v /etc/localtime:/etc/localtime:ro \
+    vuls/vuls scan \
+    -config=./config.toml # path to config.toml in docker
+```
+
+## Report
+
+```console
+$ docker run --rm -it \
+    -v ~/.ssh:/root/.ssh:ro \
+    -v $PWD:/vuls \
+    -v $PWD/vuls-log:/var/log/vuls \
+    -v /etc/localtime:/etc/localtime:ro \
+    vuls/vuls report \
+    -cvedb-path=/vuls/cve.sqlite3 \
+    -format-short-text \
+    -config=./config.toml # path to config.toml in docker
+```
+
+## tui
+
+```console
+$ docker run --rm -it \
+    -v $PWD:/vuls \
+    -v $PWD/vuls-log:/var/log/vuls \
+    vuls/vuls tui \
+    -cvedb-path=/vuls/cve.sqlite3 
+```
+
+## vulsrepo
+
+Prease refer to [this](https://hub.docker.com/r/vuls/vulsrepo/).
+
+# User Feedback
+
+## Documentation
+
+Documentation for this image is stored in the [`docker/` directory](https://github.com/future-architect/vuls/tree/master/setup/docker) of the [`future-architect/vuls` GitHub repo](https://github.com/future-architect/vuls). 
+
+## Issues
+
+If you have any problems with or questions about this image, please contact us through a [GitHub issue](https://github.com/future-architect/vuls/issues). 
+
+## Contributing
+
+1. fork a repository: github.com/future-architect/vuls to github.com/you/repo
+1. get original code: go get github.com/future-architect/vuls
+1. work on original code
+1. add remote to your repo: git remote add myfork https://github.com/you/repo.git
+1. push your changes: git push myfork
+1. create a new Pull Request

setup/docker/vulsrepo/latest/Dockerfile

@@ -0,0 +1,31 @@
+FROM httpd:2.4
+
+MAINTAINER hikachan sadayuki-matsuno
+# install packages
+RUN apt-get update \
+        && apt-get install -y --no-install-recommends \
+      	  ca-certificates \
+          vim \
+          git \
+          libcgi-pm-perl \
+          libjson-perl \
+        && rm -r /var/lib/apt/lists/*
+
+# env
+ENV HTTPD_PREFIX /usr/local/apache2
+
+VOLUME /vuls
+
+WORKDIR ${HTTPD_PREFIX}/htdocs
+RUN git clone https://github.com/usiusi360/vulsrepo.git \
+     && echo "LoadModule cgid_module modules/mod_cgid.so" >> $HTTPD_PREFIX/conf/httpd.conf \
+     && echo "<Directory \"$HTTPD_PREFIX/htdocs/vulsrepo/dist/cgi\">" >> $HTTPD_PREFIX/conf/httpd.conf \
+     && echo "  Options +ExecCGI +FollowSymLinks" >> $HTTPD_PREFIX/conf/httpd.conf \
+     && echo "  AddHandler cgi-script cgi" >> $HTTPD_PREFIX/conf/httpd.conf \
+     && echo "</Directory>" >> $HTTPD_PREFIX/conf/httpd.conf \
+     && sed -i -e 's/User daemon/#User/g' $HTTPD_PREFIX/conf/httpd.conf \
+     && sed -i -e 's/Group daemon/#Group/g' $HTTPD_PREFIX/conf/httpd.conf \
+     && ln -snf /vuls/results /usr/local/apache2/htdocs/vulsrepo/results
+
+EXPOSE 80
+CMD ["httpd-foreground"]

setup/docker/vulsrepo/latest/README.md

@@ -0,0 +1,47 @@
+# VulsRepo-Docker
+
+This is the Git repo of the official Docker image for vulsrepo.
+See the [Hub page](https://hub.docker.com/r/vuls/vulsrepo/) for the full readme on how to use the Docker image and for information regarding contributing and issues.
+
+# Supported tags and respective `Dockerfile` links
+
+- [`latest` (*vulsrepo:latest Dockerfile*)](https://github.com/future-architect/vuls/blob/master/setup/docker/vulsrepo/latest/Dockerfile)
+
+# Caution
+
+This image is built per commit.
+If you want to use the latest docker image, you should remove the existing image, and pull it once again.
+
+# What is vulsrepo?
+
+VulsRepo is visualized based on the json report output in [vuls](https://github.com/future-architect/vuls).
+
+# How to use this image
+
+## vulsrepo
+
+```console
+$docker run -dt \
+    -v $PWD:/vuls \
+    -p 80:80 \
+    vuls/vulsrepo
+```
+
+# User Feedback
+
+## Documentation
+
+Documentation for this image is stored in the [`docker/` directory](https://github.com/future-architect/vuls/tree/master/setup/docker) of the [`future-architect/vuls` GitHub repo](https://github.com/future-architect/vuls). 
+
+## Issues
+
+If you have any problems with or questions about this image, please contact us through a [GitHub issue](https://github.com/future-architect/vuls/issues). 
+
+## Contributing
+
+1. fork a repository: github.com/future-architect/vuls to github.com/you/repo
+1. get original code: go get github.com/future-architect/vuls
+1. work on original code
+1. add remote to your repo: git remote add myfork https://github.com/you/repo.git
+1. push your changes: git push myfork
+1. create a new Pull Request

util/logutil.go

@@ -18,7 +18,6 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
 package util
 
 import (
-	"fmt"
 	"os"
 	"path/filepath"
 	"runtime"
@@ -53,12 +52,7 @@ func NewCustomLogger(c config.ServerInfo) *logrus.Entry {
 
 	whereami := "localhost"
 	if 0 < len(c.ServerName) {
-		if 0 < len(c.Container.ContainerID) {
-			whereami = fmt.Sprintf(
-				"%s_%s", c.ServerName, c.Container.Name)
-		} else {
-			whereami = fmt.Sprintf("%s", c.ServerName)
-		}
+		whereami = c.GetServerName()
 	}
 
 	if _, err := os.Stat(logDir); err == nil {

util/util.go

@@ -111,7 +111,7 @@ func ProxyEnv() string {
 
 // PrependProxyEnv prepends proxy enviroment variable
 func PrependProxyEnv(cmd string) string {
-	if config.Conf.HTTPProxy == "" {
+	if len(config.Conf.HTTPProxy) == 0 {
 		return cmd
 	}
 	return fmt.Sprintf("%s %s", ProxyEnv(), cmd)
@@ -124,3 +124,14 @@ func PrependProxyEnv(cmd string) string {
 //      }
 //      return time.Unix(i, 0), nil
 //  }
+
+// Truncate truncates string to the length
+func Truncate(str string, length int) string {
+	if length < 0 {
+		return str
+	}
+	if length <= len(str) {
+		return str[:length]
+	}
+	return str
+}

util/util_test.go

@@ -131,3 +131,43 @@ func TestPrependHTTPProxyEnv(t *testing.T) {
 	}
 
 }
+
+func TestTruncate(t *testing.T) {
+	var tests = []struct {
+		in     string
+		length int
+		out    string
+	}{
+		{
+			in:     "abcde",
+			length: 3,
+			out:    "abc",
+		},
+		{
+			in:     "abcdefg",
+			length: 5,
+			out:    "abcde",
+		},
+		{
+			in:     "abcdefg",
+			length: 10,
+			out:    "abcdefg",
+		},
+		{
+			in:     "abcdefg",
+			length: 0,
+			out:    "",
+		},
+		{
+			in:     "abcdefg",
+			length: -1,
+			out:    "abcdefg",
+		},
+	}
+	for _, tt := range tests {
+		actual := Truncate(tt.in, tt.length)
+		if actual != tt.out {
+			t.Errorf("\nexpected: %s\n  actual: %s", tt.out, actual)
+		}
+	}
+}

version/version.go

@@ -1,24 +0,0 @@
-/* Vuls - Vulnerability Scanner
-Copyright (C) 2016  Future Architect, Inc. Japan.
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-package version
-
-// Name is Vuls
-const Name string = "vuls"
-
-// Version of Vuls
-const Version string = "0.1.5"