1 The short road to practicality
In the previous two blog posts, I covered some basic uses of the Bro scripting language by using it to solve large parts of a network forensics challenge. If you've read those previous posts, you'll remember that Bro scripting is an event driven language, meaning that Bro generates events based on the network traffic it observes and the scripting language can be used to apply logical processing to those events. As we saw in the previous post, Bro generates a ton of events and wading through those events to find the appropriate one is a trial that you can overcome through experience and maybe some help from grep!
Much of what I covered in the first two posts didn't have much practical application as it was intended to illustrate what you can do with Bro scripting. We were working off a trace file and a set of questions and we worked to generate a report for those questions. If there's one thing I've learned from talking to Seth Hall (One of the Bro developers and @remor on Twitter) it's that a solution has to be deployable across an enterprise to be worth your time. If your detection method includes "Open up wireshark and load the pcap" that is detection after the fact and you should find a way to make that action automated. Wireshark has its place and it does the job it was designed for very well, however, it can't be deployed across a large scale production environment like Bro can. As I've done in the past posts, I still start with tshark to help me identify the behavior I'm interested with and then pivot to Bro scripts to deploy it widescale. With Bro, we want to leverage the scripting language to be able to define activities of interest and report on them.
Much of what I covered in the first two posts didn't have much practical application as it was intended to illustrate what you can do with Bro scripting. We were working off a trace file and a set of questions and we worked to generate a report for those questions. If there's one thing I've learned from talking to Seth Hall (One of the Bro developers and @remor on Twitter) it's that a solution has to be deployable across an enterprise to be worth your time. If your detection method includes "Open up wireshark and load the pcap" that is detection after the fact and you should find a way to make that action automated. Wireshark has its place and it does the job it was designed for very well, however, it can't be deployed across a large scale production environment like Bro can. As I've done in the past posts, I still start with tshark to help me identify the behavior I'm interested with and then pivot to Bro scripts to deploy it widescale. With Bro, we want to leverage the scripting language to be able to define activities of interest and report on them.
2 Detecting web sites that use basic auth
There are still a good number of sites using Basic Access Authentication and it's likely not something you'd like to see running as a service on your network. If you've not toyed with Basic Access Authentication, it's effectively just a way to make sure that non-HTTP-compatible characters can be transmitted by using Base64 encoding. When a site uses basic auth, it sends the username and password as a colon separated string that has been Base64 encoded. While confidentiality of the username and password are not the primary intent of basic auth, I wouldn't be surprised to find a number of web developers who consider it 'secure enough' because they don't think someone is listening. To illustrate the point about Base64, here are two short snippets of ruby that will encrypt a username and password in the same way basic auth does and a short script to decrypt the Base64 string.
require 'base64' username = 'srunnels' password = 'recursivehoff' p Base64.encode64("#{username}:#{password}")
require 'base64' base64_string = 'c3J1bm5lbHM6cmVjdXJzaXZlaG9mZg==' p Base64.decode64(base64_string)
3 How it looks on the line
Since I didn't particularly want to futz with someone else's web server, I stood up a VM with Apache2 and applied Basic Auth to the default site. This way I'll have something I can regularly make requests to as well as something I can use to generate full trace file without possible exposure. With the server up and trace file being generated, we were able to capture some traffic to the site. As usual, I tend to use tshark as my initial tool, so let's find out what we're actually trying to detect.
mac@lubuntu-VM:~$ tshark -r tracefiles/20120503120402.lpc -R "http contains Auth" -O http -V | awk '/Authorization: Basic/ {print}'
Authorization: Basic c3J1bm5lbHM6cmVjdXJzaXZlaG9mZg==\r\n
Authorization: Basic c3J1bm5lbHM6cmVjdXJzaXZlaG9mZg==\r\n
Authorization: Basic c3J1bm5lbHM6cmVjdXJzaXZlaG9mZg==\r\n
Authorization: Basic c3J1bm5lbHM6cmVjdXJzaXZlaG9mZg==\r\n
The Base64 encoded string is sent to the server as part of the HTTP header, which means to start, we're going to take a look for any Bro events that correspond to http and http headers. Some easy grepping through the base scripts from Bro leads us to two events http_header() and http_all_headers(), the difference between the two being that http_header generates an event for every header while http_all_headers will generate a list of headers per request or response. For now we're going to work with http_header and see if it can detect a session using basic auth.
event http_header(c: connection, is_orig: bool, name: string, value: string)
{
if (/AUTHORIZATION/ in name && /Basic/ in value)
print fmt("%s: %s", name, value);
}
AUTHORIZATION: Basic c3J1bm5lbHM6cmVjdXJzaXZlaG9mZg== AUTHORIZATION: Basic c3J1bm5lbHM6cmVjdXJzaXZlaG9mZg== AUTHORIZATION: Basic c3J1bm5lbHM6cmVjdXJzaXZlaG9mZg== AUTHORIZATION: Basic c3J1bm5lbHM6cmVjdXJzaXZlaG9mZg==Three lines of Bro's scripting language and we can detect a server using Basic Access Authentication! Now, all that's left is to make Bro understand that we care about this kind of behavior!
4 Generating notices
Bro's Notice Framework is expansive to say the least. I found that, again, the best resource is the set of scripts that ship with Bro by default. Running a recursive grep for 'notice' in \/usr\/local\/bro\/share\/ returned 490 lines and taking a look through them, a couple entries stood out as being of possible importance. One of the common entries was "redef enum Notice::Type += {". If you're unfamiliar with the += operator, it's an operator that allows us to add onto an already defined variable. In this case we're adding a value to the enumerable constant Notice::Type. The documentation for Notice::Type lets us know that "Scripts creating new notices need to redef this enum to add their own specific notice types which would then get used when they call the NOTICE function." So, in our case we might enumerate this constant to include "HTTP::Basic_Auth_Server".
module HTTP;
export {
redef enum Notice::Type += {
## Generated if a site is detected using Basic Access Authentication
HTTP::Basic_Auth_Server
};
}
The other entry that stood out from poking around the default scripts was NOTICE(). NOTICE() takes one argument, the Notice::Info record, but it's a whopper. You can pass a rather massive amount of information into NOTICE() via the Notice::Info record but the only required argument to pass in is the Notice::Type. If we just wanted to generate a notice, albeit a somewhat unhelpful one, we could pass it just the Notice::Type we added.
module HTTP;
export {
redef enum Notice::Type += {
## Generated if a site is detected using Basic Access Authentication
HTTP::Basic_Auth_Server
};
}
event http_header(c: connection, is_orig: bool, name: string, value: string)
{
if (/AUTHORIZATION/ in name && /Basic/ in value)
{
NOTICE([$note=HTTP::Basic_Auth_Server]);
}
}
When we run the script against the tracefile, we get a notice.log in the current working directory.
#separator \x09 #set_separator , #empty_field (empty) #unset_field - #path notice #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto note msg sub src dst p n peer_descr actions policy_items suppress_for dropped remote_location.countr #types time string addr port addr port enum enum string string addr addr port count string table[enum] table[count] interval bool string string string double double addr string 1336061141.701690 - - - - - - HTTP::Basic_Auth_Server - - - - - - bro Notice::ACTION_LOG 6 3600.000000 F - - - - 1336061141.914860 - - - - - - HTTP::Basic_Auth_Server - - - - - - bro Notice::ACTION_LOG 6 3600.000000 F - - - - 1336061141.918352 - - - - - - HTTP::Basic_Auth_Server - - - - - - bro Notice::ACTION_LOG 6 3600.000000 F - - - - 1336061147.472010 - - - - - - HTTP::Basic_Auth_Server - - - - - - bro Notice::ACTION_LOG 6 3600.000000 F - - - -Like I said, a notice albeit, an uninformative one! Let's take a look at what happens when we give NOTICE() a Notice::Type and a connection.
module HTTP;
export {
redef enum Notice::Type += {
## Generated if a site is detected using Basic Access Authentication
HTTP::Basic_Auth_Server
};
}
event http_header(c: connection, is_orig: bool, name: string, value: string)
{
if (/AUTHORIZATION/ in name && /Basic/ in value)
{
NOTICE([$note=HTTP::Basic_Auth_Server,
$conn=c
]);
}
}
#separator \x09 #set_separator , #empty_field (empty) #unset_field - #path notice #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto note msg sub src dst p n peer_descr actions policy_items suppress_for dropped remote_location.countr #types time string addr port addr port enum enum string string addr addr port count string table[enum] table[count] interval bool string string string double double addr string 1336061141.701690 j931OBJ1895 192.168.164.198 51844 192.168.164.185 80 tcp HTTP::Basic_Auth_Server - - 192.168.164.198 192.168.164.185 80 - bro Notice::ACTION_LOG 6 3600.000000 1336061141.914860 j931OBJ1895 192.168.164.198 51844 192.168.164.185 80 tcp HTTP::Basic_Auth_Server - - 192.168.164.198 192.168.164.185 80 - bro Notice::ACTION_LOG 6 3600.000000 1336061141.918352 j931OBJ1895 192.168.164.198 51844 192.168.164.185 80 tcp HTTP::Basic_Auth_Server - - 192.168.164.198 192.168.164.185 80 - bro Notice::ACTION_LOG 6 3600.000000 1336061147.472010 zKNVZaX8uS7 192.168.164.198 51845 192.168.164.185 80 tcp HTTP::Basic_Auth_Server - - 192.168.164.198 192.168.164.185 80 - bro Notice::ACTION_LOG 6 3600.000000 (END)Simply adding a connection as an argument allowed Bro's Notice Framework to fill in the the uid, originator's host and port, and the responder's host and port. Now we have a notice that is of actual use! Can we make it better? I think so. There's something interesting to note in the connection that gets passed into http_header().
[id=[orig_h=192.168.164.198, orig_p=51845/tcp, resp_h=192.168.164.185, resp_p=80/tcp], orig=[size=359, state=1, num_pkts=2, num_bytes_ip=112], resp=[size=0, state=0, num_pkts=0, num_bytes_ip=0], start_time=1336061147.471671, duration=0.000339, service={^J^IHTTP^J}, addl=, hot=0, history=ScAD, uid=MBPKpZ7z43e, dpd=<uninitialized>, conn=<uninitialized>, extract_orig=F, extract_resp=F, dns=<uninitialized>, dns_state=<uninitialized>, ftp=<uninitialized>, http=[ts=1336061147.47201, uid=MBPKpZ7z43e, id=[orig_h=192.168.164.198, orig_p=51845/tcp, resp_h=192.168.164.185, resp_p=80/tcp], trans_depth=1, method=GET, host=192.168.164.185, uri=/test.html, referrer=<uninitialized>, user_agent=Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:10.0.2) Gecko/20100101 Firefox/10.0.2, request_body_len=0, response_body_len=0, status_code=<uninitialized>, status_msg=<uninitialized>, info_code=<uninitialized>, info_msg=<uninitialized>, filename=<uninitialized>, tags={^J^J}, username=srunnels, password=<uninitialized>, capture_password=F, proxied=<uninitialized>, mime_type=<uninitialized>, first_chunk=T, md5=<uninitialized>, calc_md5=F, calculating_md5=F, extraction_file=<uninitialized>, extract_file=F], http_state=[pending={^J^I[1] = [ts=1336061147.47201, uid=MBPKpZ7z43e, id=[orig_h=192.168.164.198, orig_p=51845/tcp, resp_h=192.168.164.185, resp_p=80/tcp], trans_depth=1, method=GET, host=192.168.164.185, uri=/test.html, referrer=<uninitialized>, user_agent=Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:10.0.2) Gecko/20100101 Firefox/10.0.2, request_body_len=0, response_body_len=0, status_code=<uninitialized>, status_msg=<uninitialized>, info_code=<uninitialized>, info_msg=<uninitialized>, filename=<uninitialized>, tags={^J^J^I}, username=srunnels, password=<uninitialized>, capture_password=F, proxied=<uninitialized>, mime_type=<uninitialized>, first_chunk=T, md5=<uninitialized>, calc_md5=F, calculating_md5=F, extraction_file=<uninitialized>, extract_file=F]^J}, current_request=1, current_response=0], irc=<uninitialized>, smtp=<uninitialized>, smtp_state=<uninitialized>, ssh=<uninitialized>, ssl=<uninitialized>, syslog=<uninitialized>]
It's hard to see through the massive amount of information that Bro includes, but if you look you'll notice three fields are filled in that are directly related to what we're working on.
username=srunnels, password=<uninitialized>, capture_password=F,Bro has not only already detected that username and password was passed across the line but it already decrypted it! The capture_password field is set to False by default in Bro, but it clearly got our username so Bro must have a way of decrypting base64. To the grep-mobile!
1092 ## Decodes a Base64-encoded string. 1093 ## 1094 ## s: The Base64-encoded string. 1095 ## 1096 ## Returns: The decoded version of *s*. 1097 ## 1098 ## .. bro:see:: decode_base64_custom 1099 global decode_base64: function(s: string): string;Some day, I'll stop being shocked by everything Bro does and just accept that it's wall-to-wall awesome. Well, if Bro can do it, let's make our notice include it as well!
module HTTP;
export {
redef enum Notice::Type += {
## Generated if a site is detected using Basic Access Authentication
HTTP::Basic_Auth_Server
};
}
event http_header(c: connection, is_orig: bool, name: string, value: string)
{
if (/AUTHORIZATION/ in name && /Basic/ in value)
{
local parts: string_array;
parts = split1(decode_base64(sub_bytes(value, 7, |value|)), /:/);
NOTICE([$note=HTTP::Basic_Auth_Server,
$msg=fmt("username: %s password: %s", parts[1],
HTTP::default_capture_password == F ? "Blocked" : parts[2]),
$conn=c
]);
}
}
Here, I've taken the value we find by checking for Authorization Basic in the header and break it into pieces using sub_bytes() and split1(). The function sub_bytes() will take a value, a position, and a length and return a string while split1() will take a string and a regexp and output a string_array whose members are the parts of the string once they've been split one time. In this case, I've used sub_bytes() to remove the "Basic " part of the string in value, then used split1() to break the resulting string on the colon. You'll also notice that I included a ternary operator as part of the fmt() operator. I'm of the opinion that if the Bro developers did it, we should follow suit and in this case, we're only going to include the password if HTTP::default_capture_password is true. Once we run the script against our trace file, we get the output we expected in notice.log.
1336061141.701690 MzLXbKyksE9 192.168.164.198 51844 192.168.164.185 80 tcp HTTP::Basic_Auth_Server username: srunnels password: Blocked - 192.168.164.198 192.168.164.185 80 - bro Notice::ACTION_LOG 6 3600.000000 F - - - - - - - - 1336061141.914860 MzLXbKyksE9 192.168.164.198 51844 192.168.164.185 80 tcp HTTP::Basic_Auth_Server username: srunnels password: Blocked - 192.168.164.198 192.168.164.185 80 - bro Notice::ACTION_LOG 6 3600.000000 F - - - - - - - - 1336061141.918352 MzLXbKyksE9 192.168.164.198 51844 192.168.164.185 80 tcp HTTP::Basic_Auth_Server username: srunnels password: Blocked - 192.168.164.198 192.168.164.185 80 - bro Notice::ACTION_LOG 6 3600.000000 F - - - - - - - - 1336061147.472010 sO6OVrkv5G6 192.168.164.198 51845 192.168.164.185 80 tcp HTTP::Basic_Auth_Server username: srunnels password: Blocked - 192.168.164.198 192.168.164.185 80 - bro Notice::ACTION_LOG 6 3600.000000 F - - - - - - - -If we include a command line directive to set HTTP::default_capture_password the decrypted password will be included in the notice.log file.
/usr/local/bro/bin/bro -r ~/tracefiles/20120503120402.lpc post3.bro "HTTP::default_capture_password = T;"
1336061141.701690 7VFRK2sOUl4 192.168.164.198 51844 192.168.164.185 80 tcp HTTP::Basic_Auth_Server username: srunnels password: recursivehoff - 192.168.164.198 192.168.164.185 80 - bro Notice::ACTION_LOG 6 3600.000000 F - - - - - - - - 1336061141.914860 7VFRK2sOUl4 192.168.164.198 51844 192.168.164.185 80 tcp HTTP::Basic_Auth_Server username: srunnels password: recursivehoff - 192.168.164.198 192.168.164.185 80 - bro Notice::ACTION_LOG 6 3600.000000 F - - - - - - - - 1336061141.918352 7VFRK2sOUl4 192.168.164.198 51844 192.168.164.185 80 tcp HTTP::Basic_Auth_Server username: srunnels password: recursivehoff - 192.168.164.198 192.168.164.185 80 - bro Notice::ACTION_LOG 6 3600.000000 F - - - - - - - - 1336061147.472010 JDdZhg1kTol 192.168.164.198 51845 192.168.164.185 80 tcp HTTP::Basic_Auth_Server username: srunnels password: recursivehoff - 192.168.164.198 192.168.164.185 80 - bro Notice::ACTION_LOG 6 3600.000000 F - - - - - - - -
5 Making our script more operationaly relevant
At the moment, our Bro script generates a notice every time it sees a header with the "Authorization Basic" set. For a long HTTP session, this could get ugly in the notice.log file. As well, an Incident Responder is likely to care about how many times a request was sent, only that a server using basic auth was contacted. The Notice::Info record sent to NOTICE() can include a $identifier field that is used by the notice framework to detect when a duplicate notice has been created. For our identifier well use the responder's IP address and port to create the identifier. We're also going to set the $suppress_for argument to indicate how long the alert should be suppressed which will give us a HTTP::Basic_Auth_Server notice once per server per day.
module HTTP;
export {
redef enum Notice::Type += {
## Generated if a site is detected using Basic Access Authentication
HTTP::Basic_Auth_Server
};
}
redef Notice::Policy += {
[$pred(n: Notice::Info) =
{
return n$note == HTTP::Basic_Auth_Server && Site::is_local_addr(n$id$resp_h);
},
$action = Notice::ACTION_EMAIL
]
};
event http_header(c: connection, is_orig: bool, name: string, value: string)
{
if (/AUTHORIZATION/ in name && /Basic/ in value)
{
local parts: string_array;
parts = split1(decode_base64(sub_bytes(value, 7, |value|)), /:/);
if (|parts| == 2)
NOTICE([$note=HTTP::Basic_Auth_Server,
$msg=fmt("username: %s", parts[1]),
$identifier=cat(c$id$resp_h, c$id$resp_p),
$suppress_for=1day,
$conn=c
]);
}
}
Now, when we run the script against the tracefile we only get one notice in notice.log.
1336061141.701690 ZI9LtsaV4Qh 192.168.164.198 51844 192.168.164.185 80 tcp HTTP::Basic_Auth username: srunnels password: recursivehoff - 192.168.164.198 192.168.164.185 80 - bro Notice::ACTION_LOG 6 86400.000000 F - - - - - - - -Another utility we might like to add is to detect when the server using basic auth is one of our servers. To tell Bro to send an email when a notice is generated, we need to write a Notice::policy item that includes an action ($action) and an anonymous function used to determine if sending the email is appropriate ($pred). The appropriate $action would be Notice::ACTION_EMAIL and the predicate depends on the situation and how you want to restrict it. For detecting sites using basic auth, logging a notice to file is fine for when a local system access a server using basic auth, but it could be pretty handy to receive an email if the server using basic auth turns out to be local. So for our script we'd use a Notice::policy like:
redef Notice::policy += {
[$pred(n: Notice::Info) =
{
return n$note == HTTP::Basic_Auth_Server && Site::is_local_addr(n$id$resp_h);
}
$action = Notice::ACTION_EMAIL
]
};
Here, we've created an anonymous function that returns true or false depending on if our note is HTTP::Basic_Auth_Server and the host of the responder is determined to be local. If the $pred returns True, the $action is taken resulting in an email alert being sent.
6 Wrapping up
Bro is surprisingly complex. There is so much going on beneath the hood, that I'm not sure I'll ever fully understand it - especially given how quickly the dev's move. That being said, I believe the dev's have done a superb job making sure the users have the tools they need at hand. Making heavy use of the online documentation and simply searching through the default scripts shipped with Bro can bring forth a massive learning opportunity! The fact that you can go from evidence to writing scripts that generate notices in a short period of time let's you make "closing the loop" part of your regular incident response cycle.
7 Repository
If you're interested in running the script built in this blog post they are posted in a github repository.
To include this in your current build of Bro, follow the commands below.
cd /usr/local/bro/share/bro/site git clone git://github.com/srunnels/bro-scripts srunnels-scripts echo "@load srunnels-scripts/http-basic-auth" >> local.bro # If you want to receieve emails about local basic auth notices echo "@load srunnels-scripts/notice-handling" >> local.bro
Then issue "install" and "restart" from within broctl.
Date: 2012/05/03
Date: 2012/05/03
Org version 7.8.06 with Emacs version 24
Validate XHTML 1.0
