feat(libscan): enable to scan vulns of libs with pseudo #1035 (#1050)

* finished to implement new mail client

* delete email_test.go

README.md

@@ -9,7 +9,7 @@
 
 ![Vuls-logo](img/vuls_logo.png)
 
-Vulnerability scanner for Linux/FreeBSD, agentless, written in golang.
+Vulnerability scanner for Linux/FreeBSD, agent-less, written in Go.
 We have a slack team. [Join slack team](http://goo.gl/forms/xm5KFo35tu)
 Twitter: [@vuls_en](https://twitter.com/vuls_en)
 
@@ -23,20 +23,6 @@ Twitter: [@vuls_en](https://twitter.com/vuls_en)
 
 ----
 
-## NEWS
-
-| Version     | Main Feature |  Date |
-|:------------|:---------------------------------|:--------------------|
-| [v0.8.0](https://github.com/future-architect/vuls/releases/tag/v0.8.0) | secret | Coming soon |
-| [v0.7.0](https://github.com/future-architect/vuls/releases/tag/v0.7.0) | WordPress Vulnerability Scan | 2019/Apr/8 |
-| [v0.6.3](https://github.com/future-architect/vuls/releases/tag/v0.6.3) | GitHub Integration | 2019/Feb/20 |
-| [v0.6.2](https://github.com/future-architect/vuls/releases/tag/v0.6.2) | Add US-CERT/JPCERT Alerts as VulnSrc | 2019/Jan/23 |
-| [v0.6.1](https://github.com/future-architect/vuls/releases/tag/v0.6.1) | BugFix | 2018/Nov/16 |
-| [v0.6.0](https://github.com/future-architect/vuls/releases/tag/v0.6.0) | Add ExploitDB as VulnSrc | 2018/Nov/3 |
-| [v0.5.0](https://github.com/future-architect/vuls/releases/tag/v0.5.0) | Scan accuracy improvement | 2018/Aug/27 |
-
-----
-
 ## Abstract
 
 For a system administrator, having to perform security vulnerability analysis and software update on a daily basis can be a burden.
@@ -66,36 +52,47 @@ Vuls is a tool created to solve the problems listed above. It has the following
 
 - Alpine, Amazon Linux, CentOS, Debian, Oracle Linux, Raspbian, RHEL, SUSE Enterprise Linux, and Ubuntu
 - FreeBSD
-- Cloud, on-premise, Docker Container and Docker Image
+- Cloud, on-premise, Running Docker Container
 
 ### High-quality scan
 
-Vuls uses multiple vulnerability databases
+- Vulnerability Database
+  - [NVD](https://nvd.nist.gov/)
+  - [JVN(Japanese)](http://jvndb.jvn.jp/apis/myjvn/)
 
-- [NVD](https://nvd.nist.gov/)
-- [JVN(Japanese)](http://jvndb.jvn.jp/apis/myjvn/)
 - OVAL
+  - [Red Hat](https://www.redhat.com/security/data/oval/)
   - [Debian](https://www.debian.org/security/oval/)
-  - [Oracle Linux](https://linux.oracle.com/security/oval/)
-  - [RedHat](https://www.redhat.com/security/data/oval/)
-  - [SUSE](http://ftp.suse.com/pub/projects/security/oval/)
   - [Ubuntu](https://people.canonical.com/~ubuntu-security/oval/)
+  - [SUSE](http://ftp.suse.com/pub/projects/security/oval/)
+  - [Oracle Linux](https://linux.oracle.com/security/oval/)
 
-- [Alpine-secdb](https://git.alpinelinux.org/cgit/alpine-secdb/)
-- [Debian Security Bug Tracker](https://security-tracker.debian.org/tracker/)
-- [Red Hat Security Advisories](https://access.redhat.com/security/security-updates/)
-- Commands (yum, zypper, and pkg-audit)
-  - RHSA/ALAS/ELSA/FreeBSD-SA
-- [Exploit Database](https://www.exploit-db.com/)
-- [US-CERT](https://www.us-cert.gov/ncas/alerts)
-- [JPCERT](http://www.jpcert.or.jp/at/2019.html)
-- [WPVulnDB](https://wpvulndb.com/api)
-- [Node.js Security Working Group](https://github.com/nodejs/security-wg)
-- [Ruby Advisory Database](https://github.com/rubysec/ruby-advisory-db)
-- [Safety DB(Python)](https://github.com/pyupio/safety-db)
-- [PHP Security Advisories Database](https://github.com/FriendsOfPHP/security-advisories)
-- [RustSec Advisory Database](https://github.com/RustSec/advisory-db)
-- Changelog
+- Security Advisory
+  - [Alpine-secdb](https://git.alpinelinux.org/cgit/alpine-secdb/)
+  - [Red Hat Security Advisories](https://access.redhat.com/security/security-updates/)
+  - [Debian Security Bug Tracker](https://security-tracker.debian.org/tracker/)
+
+- Commands(yum, zypper, pkg-audit)
+  - RHSA / ALAS / ELSA / FreeBSD-SA
+  - Changelog
+
+- PoC, Exploit
+  - [Exploit Database](https://www.exploit-db.com/)
+  - [Metasploit-Framework modules](https://www.rapid7.com/db/?q=&type=metasploit)
+
+- CERT
+  - [US-CERT](https://www.us-cert.gov/ncas/alerts)
+  - [JPCERT](http://www.jpcert.or.jp/at/2019.html)
+
+- Libraries
+  - [Node.js Security Working Group](https://github.com/nodejs/security-wg)
+  - [Ruby Advisory Database](https://github.com/rubysec/ruby-advisory-db)
+  - [Safety DB(Python)](https://github.com/pyupio/safety-db)
+  - [PHP Security Advisories Database](https://github.com/FriendsOfPHP/security-advisories)
+  - [RustSec Advisory Database](https://github.com/RustSec/advisory-db)
+
+- WordPress
+  - [WPVulnDB](https://wpvulndb.com/api)
 
 ### Scan mode
 
@@ -134,19 +131,6 @@ Vuls uses multiple vulnerability databases
 - It is possible to acquire the state of the server by connecting via SSH and executing the command.
 - Vuls warns when the scan target server was updated the kernel etc. but not restarting it.
 
-### **Static** Analysis
-
-**Image scan function is no longer supported from Vuls v0.9.5. Use Trivy directry**
-
-~~Vuls v0.8.0 can scan Docker images using [knqyf263/trivy](https://github.com/knqyf263/trivy).
-Following Registry supported.~~
-
-- ~~ECR~~
-- ~~GCR~~
-- ~~Local Image~~
-
-~~For details, see [Scan docker image](https://vuls.io/docs/en/tutorial-scan-docker-image.html)~~  
-
 ### Scan vulnerabilities of non-OS-packages
 
 - Libraries of programming language
@@ -184,7 +168,7 @@ Vuls has some options to detect the vulnerabilities
 
 ## Document
 
-For more information such as Installation, Tutorial, Usage, visit [vuls.io](https://vuls.io/)
+For more information such as Installation, Tutorial, Usage, visit [vuls.io](https://vuls.io/)  
 [日本語翻訳ドキュメント](https://vuls.io/ja/)
 
 ----
@@ -195,12 +179,6 @@ kotakanbe ([@kotakanbe](https://twitter.com/kotakanbe)) created vuls and [these
 
 ----
 
-## Change Log
-
-Please see [CHANGELOG](https://github.com/future-architect/vuls/blob/master/CHANGELOG.md).
-
-----
-
 ## Stargazers over time
 
 [![Stargazers over time](https://starcharts.herokuapp.com/future-architect/vuls.svg)](https://starcharts.herokuapp.com/future-architect/vuls)

go.mod

@@ -19,6 +19,8 @@ require (
 	github.com/boltdb/bolt v1.3.1
 	github.com/cenkalti/backoff v2.2.1+incompatible
 	github.com/d4l3k/messagediff v1.2.2-0.20190829033028-7e0a312ae40b
+	github.com/emersion/go-sasl v0.0.0-20200509203442-7bfe0ed36a21
+	github.com/emersion/go-smtp v0.13.0
 	github.com/google/subcommands v1.2.0
 	github.com/gosuri/uitable v0.0.4
 	github.com/hashicorp/go-uuid v1.0.2

go.sum

@@ -187,6 +187,10 @@ github.com/elazarl/goproxy v0.0.0-20190711103511-473e67f1d7d2/go.mod h1:/Zj4wYkg
 github.com/elazarl/goproxy/ext v0.0.0-20190421051319-9d40249d3c2f/go.mod h1:gNh8nYJoAm43RfaxurUnxr+N1PwuFV3ZMl/efxlIlY8=
 github.com/elazarl/goproxy/ext v0.0.0-20190711103511-473e67f1d7d2 h1:dWB6v3RcOy03t/bUadywsbyrQwCqZeNIEX6M1OtSZOM=
 github.com/elazarl/goproxy/ext v0.0.0-20190711103511-473e67f1d7d2/go.mod h1:gNh8nYJoAm43RfaxurUnxr+N1PwuFV3ZMl/efxlIlY8=
+github.com/emersion/go-sasl v0.0.0-20200509203442-7bfe0ed36a21 h1:OJyUGMJTzHTd1XQp98QTaHernxMYzRaOasRir9hUlFQ=
+github.com/emersion/go-sasl v0.0.0-20200509203442-7bfe0ed36a21/go.mod h1:iL2twTeMvZnrg54ZoPDNfJaJaqy0xIQFuBdrLsmspwQ=
+github.com/emersion/go-smtp v0.13.0 h1:aC3Kc21TdfvXnuJXCQXuhnDXUldhc12qME/S7Y3Y94g=
+github.com/emersion/go-smtp v0.13.0/go.mod h1:qm27SGYgoIPRot6ubfQ/GpiPy/g3PaZAVRxiO/sDUgQ=
 github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs=
 github.com/emicklei/go-restful v2.9.5+incompatible/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs=
 github.com/emirpasic/gods v1.9.0/go.mod h1:YfzfFFoVP/catgzJb4IKIqXjX78Ha8FMSDh3ymbK86o=

report/email.go

@@ -5,10 +5,11 @@ import (
 	"fmt"
 	"net"
 	"net/mail"
-	"net/smtp"
 	"strings"
 	"time"
 
+	sasl "github.com/emersion/go-sasl"
+	smtp "github.com/emersion/go-smtp"
 	"github.com/future-architect/vuls/config"
 	"github.com/future-architect/vuls/models"
 	"golang.org/x/xerrors"
@@ -21,7 +22,6 @@ func (w EMailWriter) Write(rs ...models.ScanResult) (err error) {
 	conf := config.Conf
 	var message string
 	sender := NewEMailSender()
-
 	m := map[string]int{}
 	for _, r := range rs {
 		if conf.FormatOneEMail {
@@ -85,37 +85,50 @@ type EMailSender interface {
 
 type emailSender struct {
 	conf config.SMTPConf
-	send func(string, smtp.Auth, string, []string, []byte) error
 }
 
-func smtps(emailConf config.SMTPConf, message string) (err error) {
-	auth := smtp.PlainAuth("",
-		emailConf.User,
-		emailConf.Password,
-		emailConf.SMTPAddr,
-	)
-
+func (e *emailSender) sendMail(smtpServerAddr, message string) (err error) {
+	var c *smtp.Client
+	var auth sasl.Client
+	emailConf := e.conf
 	//TLS Config
 	tlsConfig := &tls.Config{
 		ServerName: emailConf.SMTPAddr,
 	}
-
-	smtpServer := net.JoinHostPort(emailConf.SMTPAddr, emailConf.SMTPPort)
-	//New TLS connection
-	con, err := tls.Dial("tcp", smtpServer, tlsConfig)
-	if err != nil {
-		return xerrors.Errorf("Failed to create TLS connection: %w", err)
+	switch emailConf.SMTPPort {
+	case "465":
+		//New TLS connection
+		c, err = smtp.DialTLS(smtpServerAddr, tlsConfig)
+		if err != nil {
+			return xerrors.Errorf("Failed to create TLS connection to SMTP server: %w", err)
+		}
+	default:
+		c, err = smtp.Dial(smtpServerAddr)
+		if err != nil {
+			return xerrors.Errorf("Failed to create connection to SMTP server: %w", err)
+		}
 	}
-	defer con.Close()
+	defer c.Close()
 
-	c, err := smtp.NewClient(con, emailConf.SMTPAddr)
-	if err != nil {
-		return xerrors.Errorf("Failed to create new client: %w", err)
+	if err = c.Hello("localhost"); err != nil {
+		return xerrors.Errorf("Failed to send Hello command: %w", err)
 	}
+
+	if ok, _ := c.Extension("STARTTLS"); ok {
+		if err := c.StartTLS(tlsConfig); err != nil {
+			return xerrors.Errorf("Failed to STARTTLS: %w", err)
+		}
+	}
+
+	if ok, param := c.Extension("AUTH"); ok {
+		authList := strings.Split(param, " ")
+		auth = e.newSaslClient(authList)
+	}
+
 	if err = c.Auth(auth); err != nil {
 		return xerrors.Errorf("Failed to authenticate: %w", err)
 	}
-	if err = c.Mail(emailConf.From); err != nil {
+	if err = c.Mail(emailConf.From, nil); err != nil {
 		return xerrors.Errorf("Failed to send Mail command: %w", err)
 	}
 	for _, to := range emailConf.To {
@@ -169,38 +182,13 @@ func (e *emailSender) Send(subject, body string) (err error) {
 	smtpServer := net.JoinHostPort(emailConf.SMTPAddr, emailConf.SMTPPort)
 
 	if emailConf.User != "" && emailConf.Password != "" {
-		switch emailConf.SMTPPort {
-		case "465":
-			err := smtps(emailConf, message)
-			if err != nil {
-				return xerrors.Errorf("Failed to send emails: %w", err)
-			}
-		default:
-			err = e.send(
-				smtpServer,
-				smtp.PlainAuth(
-					"",
-					emailConf.User,
-					emailConf.Password,
-					emailConf.SMTPAddr,
-				),
-				emailConf.From,
-				mailAddresses,
-				[]byte(message),
-			)
-			if err != nil {
-				return xerrors.Errorf("Failed to send emails: %w", err)
-			}
+		err = e.sendMail(smtpServer, message)
+		if err != nil {
+			return xerrors.Errorf("Failed to send emails: %w", err)
 		}
 		return nil
 	}
-	err = e.send(
-		smtpServer,
-		nil,
-		emailConf.From,
-		mailAddresses,
-		[]byte(message),
-	)
+	err = e.sendMail(smtpServer, message)
 	if err != nil {
 		return xerrors.Errorf("Failed to send emails: %w", err)
 	}
@@ -209,5 +197,19 @@ func (e *emailSender) Send(subject, body string) (err error) {
 
 // NewEMailSender creates emailSender
 func NewEMailSender() EMailSender {
-	return &emailSender{config.Conf.EMail, smtp.SendMail}
+	return &emailSender{config.Conf.EMail}
+}
+
+func (e *emailSender) newSaslClient(authList []string) sasl.Client {
+	for _, v := range authList {
+		switch v {
+		case "PLAIN":
+			auth := sasl.NewPlainClient("", e.conf.User, e.conf.Password)
+			return auth
+		case "LOGIN":
+			auth := sasl.NewLoginClient(e.conf.User, e.conf.Password)
+			return auth
+		}
+	}
+	return nil
 }

report/email_test.go

@@ -1,115 +0,0 @@
-package report
-
-import (
-	"net/smtp"
-	"reflect"
-	"strings"
-	"testing"
-
-	"github.com/future-architect/vuls/config"
-)
-
-type emailRecorder struct {
-	addr string
-	auth smtp.Auth
-	from string
-	to   []string
-	body string
-}
-
-type mailTest struct {
-	in  config.SMTPConf
-	out emailRecorder
-}
-
-var mailTests = []mailTest{
-	{
-		config.SMTPConf{
-			SMTPAddr: "127.0.0.1",
-			SMTPPort: "25",
-
-			From: "from@address.com",
-			To:   []string{"to@address.com"},
-			Cc:   []string{"cc@address.com"},
-		},
-		emailRecorder{
-			addr: "127.0.0.1:25",
-			auth: smtp.PlainAuth("", "", "", "127.0.0.1"),
-			from: "from@address.com",
-			to:   []string{"to@address.com", "cc@address.com"},
-			body: "body",
-		},
-	},
-	{
-		config.SMTPConf{
-			SMTPAddr: "127.0.0.1",
-			SMTPPort: "25",
-
-			User:     "vuls",
-			Password: "password",
-
-			From: "from@address.com",
-			To:   []string{"to1@address.com", "to2@address.com"},
-			Cc:   []string{"cc1@address.com", "cc2@address.com"},
-		},
-		emailRecorder{
-			addr: "127.0.0.1:25",
-			auth: smtp.PlainAuth(
-				"",
-				"vuls",
-				"password",
-				"127.0.0.1",
-			),
-			from: "from@address.com",
-			to: []string{"to1@address.com", "to2@address.com",
-				"cc1@address.com", "cc2@address.com"},
-			body: "body",
-		},
-	},
-}
-
-func TestSend(t *testing.T) {
-	for i, test := range mailTests {
-		f, r := mockSend(nil)
-		sender := &emailSender{conf: test.in, send: f}
-
-		subject := "subject"
-		body := "body"
-		if err := sender.Send(subject, body); err != nil {
-			t.Errorf("unexpected error: %s", err)
-		}
-
-		if r.addr != test.out.addr {
-			t.Errorf("#%d: wrong 'addr' field.\r\nexpected: %s\n got: %s", i, test.out.addr, r.addr)
-		}
-
-		if !reflect.DeepEqual(r.auth, test.out.auth) && r.auth != nil {
-			t.Errorf("#%d: wrong 'auth' field.\r\nexpected: %v\n got: %v", i, test.out.auth, r.auth)
-		}
-
-		if r.from != test.out.from {
-			t.Errorf("#%d: wrong 'from' field.\r\nexpected: %v\n got: %v", i, test.out.from, r.from)
-		}
-
-		if !reflect.DeepEqual(r.to, test.out.to) {
-			t.Errorf("#%d: wrong 'to' field.\r\nexpected: %v\n got: %v", i, test.out.to, r.to)
-		}
-
-		if r.body != test.out.body {
-			t.Errorf("#%d: wrong 'body' field.\r\nexpected: %v\n got: %v", i, test.out.body, r.body)
-		}
-
-	}
-
-}
-
-func mockSend(errToReturn error) (func(string, smtp.Auth, string, []string, []byte) error, *emailRecorder) {
-	r := new(emailRecorder)
-	return func(addr string, a smtp.Auth, from string, to []string, msg []byte) error {
-		// Split into header and body
-		messages := strings.Split(string(msg), "\r\n\r\n")
-		body := messages[1]
-		*r = emailRecorder{addr, a, from, to, body}
-		return errToReturn
-	}, r
-}

scan/base.go

@@ -4,6 +4,7 @@ import (
 	"bufio"
 	"encoding/json"
 	"fmt"
+	"io/ioutil"
 	"net"
 	"os"
 	"regexp"
@@ -562,12 +563,23 @@ func (l *base) scanLibraries() (err error) {
 		if _, ok := libFilemap[path]; ok {
 			continue
 		}
-		cmd := fmt.Sprintf("cat %s", path)
-		r := exec(l.ServerInfo, cmd, noSudo)
-		if !r.isSuccess() {
-			return xerrors.Errorf("Failed to get target file: %s, filepath: %s", r, path)
+
+		var bytes []byte
+		switch l.Distro.Family {
+		case config.ServerTypePseudo:
+			bytes, err = ioutil.ReadFile(path)
+			if err != nil {
+				return xerrors.Errorf("Failed to get target file: %s, filepath: %s", err, path)
+			}
+		default:
+			cmd := fmt.Sprintf("cat %s", path)
+			r := exec(l.ServerInfo, cmd, noSudo)
+			if !r.isSuccess() {
+				return xerrors.Errorf("Failed to get target file: %s, filepath: %s", r, path)
+			}
+			bytes = []byte(r.Stdout)
 		}
-		libFilemap[path] = []byte(r.Stdout)
+		libFilemap[path] = bytes
 	}
 
 	for path, b := range libFilemap {