pingpong.git
4 years agoAdd getPackets to Conversation's API.
Janus Varmarken [Fri, 4 May 2018 00:02:11 +0000 (17:02 -0700)]
Add getPackets to Conversation's API.

4 years agoExtract Conversation to separate file (i.e. it is no longer an inner class). Add...
Janus Varmarken [Thu, 3 May 2018 23:41:59 +0000 (16:41 -0700)]
Extract Conversation to separate file (i.e. it is no longer an inner class). Add List of associated packets to Conversation class. Add Javadoc. Reformat variable names to match android convetions (indicating member variables by prefixing these with 'm').

4 years agoAdding pcap files for remote ON and remote ON charging cases.
rtrimana [Wed, 2 May 2018 23:58:50 +0000 (16:58 -0700)]
Adding pcap files for remote ON and remote ON charging cases.

4 years agoAdding combined PCAP (local and remote) to test out the algorithm- it successfully...
rtrimana [Wed, 2 May 2018 23:10:24 +0000 (16:10 -0700)]
Adding combined PCAP (local and remote) to test out the algorithm- it successfully detects the only pattern specified in the FlowPattern class.

4 years agoSeparating pattern collection and analysis into 2 different threads.
rtrimana [Wed, 2 May 2018 23:04:43 +0000 (16:04 -0700)]
Separating pattern collection and analysis into 2 different threads.

4 years agoMaking retransmission check O(1) using HashSet/Set.
rtrimana [Wed, 2 May 2018 00:41:59 +0000 (17:41 -0700)]
Making retransmission check O(1) using HashSet/Set.

4 years agoRefactoring and restructuring - Adding DnsMap class
rtrimana [Tue, 1 May 2018 23:53:21 +0000 (16:53 -0700)]
Refactoring and restructuring - Adding DnsMap class

4 years agoFilter out retransmissions when reconstructing TCP flows. Now able to detect all...
Janus Varmarken [Sun, 29 Apr 2018 07:49:12 +0000 (00:49 -0700)]
Filter out retransmissions when reconstructing TCP flows. Now able to detect all local ON events in wlan1.local.dns.pcap.

4 years agoUpdate pcap4j to v2.0.0-alpha to get access to packet timestamps (and possibly TCP...
Janus Varmarken [Sun, 29 Apr 2018 03:34:58 +0000 (20:34 -0700)]
Update pcap4j to v2.0.0-alpha to get access to packet timestamps (and possibly TCP session reassembly at a later stage). Now prints the timestamp for the occurrence of a complete match in the trace.

4 years ago1) Skip zero-payload packets when reassemlbing conversations from individual packets...
Janus Varmarken [Sun, 29 Apr 2018 03:01:20 +0000 (20:01 -0700)]
1) Skip zero-payload packets when reassemlbing conversations from individual packets. 2) Hardcode TP-Link Local ON pattern. 3) Rushed implementation that finds complete matches of the pattern.

4 years agoFirst small step towards pattern search: separate packets related to packet into...
Janus Varmarken [Sun, 29 Apr 2018 01:46:40 +0000 (18:46 -0700)]
First small step towards pattern search: separate packets related to packet into separate lists, one list for each conversation/session.

4 years agoConverted IP to hostname map from Map<String,List<String>> to Map<String,Set<String...
Janus Varmarken [Fri, 27 Apr 2018 23:56:44 +0000 (16:56 -0700)]
Converted IP to hostname map from Map<String,List<String>> to Map<String,Set<String>> to prevent multiple entries of the same hostname showing up for the same IP. Added some javadoc. Removed KataiStruct packet representations.

4 years agoAdd functional code that loads a pcap file and constructs the IP->hostname map/dictio...
Janus Varmarken [Fri, 27 Apr 2018 23:37:00 +0000 (16:37 -0700)]
Add functional code that loads a pcap file and constructs the IP->hostname map/dictionary. Note that the map is actually of type Map<String, List<String>> as the trace contains cases where ONE IP maps to MULTIPLE hostnames. Bug in current implementation: the map should be changed to Map<String, Set<String>> to prevent the same hostname from showing up in the list multiple times (occurs when a set of DNS queries return the same IP).

4 years agoKaitai parser half baked; there seems to be a problem with the parsing of DNS packets...
rtrimana [Thu, 26 Apr 2018 22:31:26 +0000 (15:31 -0700)]
Kaitai parser half baked; there seems to be a problem with the parsing of DNS packets (missing type A addresses in the data structure).

4 years agoManaged to pick and parse DNS packets; but, still need to get the detailed informatio...
rtrimana [Thu, 26 Apr 2018 17:52:01 +0000 (10:52 -0700)]
Managed to pick and parse DNS packets; but, still need to get the detailed information from inside the packet.

4 years agoReading and parsing through packets; handling unwanted packets; ready to create a...
rtrimana [Thu, 26 Apr 2018 00:42:55 +0000 (17:42 -0700)]
Reading and parsing through packets; handling unwanted packets; ready to create a good parser to create a nice data structure.

4 years agoSkipping packets that do not have etherType, e.g. XID, EAPOL, etc.
rtrimana [Wed, 25 Apr 2018 18:55:48 +0000 (11:55 -0700)]
Skipping packets that do not have etherType, e.g. XID, EAPOL, etc.

4 years agoFixing build flow (broken because of a missing return statement).
rtrimana [Wed, 25 Apr 2018 01:00:55 +0000 (18:00 -0700)]
Fixing build flow (broken because of a missing return statement).

4 years agoResolving merge conflict in Main.java
rtrimana [Wed, 25 Apr 2018 00:51:26 +0000 (17:51 -0700)]
Resolving merge conflict in Main.java

4 years agoAdding the Kaitai library to parse PCAP files.
rtrimana [Wed, 25 Apr 2018 00:49:45 +0000 (17:49 -0700)]
Adding the Kaitai library to parse PCAP files.

4 years agoadd notes
Janus Varmarken [Tue, 24 Apr 2018 18:04:38 +0000 (11:04 -0700)]
add notes

4 years agoapply application plugin
Janus Varmarken [Tue, 24 Apr 2018 16:48:07 +0000 (09:48 -0700)]
apply application plugin

4 years agoadd project for Smart Plug detection
Janus Varmarken [Tue, 24 Apr 2018 16:30:40 +0000 (09:30 -0700)]
add project for Smart Plug detection

4 years agoUpdating coloring for graphs.
rtrimana [Fri, 23 Mar 2018 15:17:04 +0000 (08:17 -0700)]
Updating coloring for graphs.

4 years agoChanging Packet Bytes into Traffic Volume in the generated time series graphs.
rtrimana [Wed, 21 Mar 2018 17:25:56 +0000 (10:25 -0700)]
Changing Packet Bytes into Traffic Volume in the generated time series graphs.

4 years agoMore scripts to plot graphs with labels.
rtrimana [Mon, 19 Mar 2018 22:36:55 +0000 (15:36 -0700)]
More scripts to plot graphs with labels.

4 years agoSIGCOMM paper template
Janus Varmarken [Sat, 10 Mar 2018 07:51:47 +0000 (23:51 -0800)]
SIGCOMM paper template

4 years agoAdjustments for the addition of eth1 for SmartThings plug
rtrimana [Fri, 23 Feb 2018 18:59:09 +0000 (10:59 -0800)]
Adjustments for the addition of eth1 for SmartThings plug

4 years agoFixing G.nodes() iteration that needs us to force Python to create a copy for iterati...
rtrimana [Wed, 21 Feb 2018 19:40:23 +0000 (11:40 -0800)]
Fixing G.nodes() iteration that needs us to force Python to create a copy for iteration condition by adding list()

4 years agoRemoving main_flow.sh; fixing flow to take time series plots from non DNS data
rtrimana [Thu, 15 Feb 2018 19:52:44 +0000 (11:52 -0800)]
Removing main_flow.sh; fixing flow to take time series plots from non DNS data

4 years agoWorking scripts and plots for 4 devices (smart plugs)
rtrimana [Wed, 14 Feb 2018 19:45:27 +0000 (11:45 -0800)]
Working scripts and plots for 4 devices (smart plugs)

4 years agoAdding combining plots script
rtrimana [Fri, 9 Feb 2018 19:39:44 +0000 (11:39 -0800)]
Adding combining plots script

4 years agoCompleting a new flow that will do automated analysis and graph generation for one...
rtrimana [Wed, 7 Feb 2018 23:58:03 +0000 (15:58 -0800)]
Completing a new flow that will do automated analysis and graph generation for one device

4 years agoAdding a new flow to yield per device data, analysis, and graphs.
rtrimana [Tue, 6 Feb 2018 23:40:26 +0000 (15:40 -0800)]
Adding a new flow to yield per device data, analysis, and graphs.

4 years agoRemoving bipartite_iot_web_gexf_generator.py; the bipartite feature has been merged...
rtrimana [Fri, 2 Feb 2018 22:00:00 +0000 (14:00 -0800)]
Removing bipartite_iot_web_gexf_generator.py; the bipartite feature has been merged into base_gexf_generator.py

4 years agoMerging bipartite functionality into the main base_gexf_generator.py script
rtrimana [Fri, 2 Feb 2018 17:25:21 +0000 (09:25 -0800)]
Merging bipartite functionality into the main base_gexf_generator.py script

4 years agoMerge branch 'master' of https://github.uci.edu/rtrimana/smart_home_traffic
rtrimana [Fri, 1 Dec 2017 23:36:30 +0000 (15:36 -0800)]
Merge branch 'master' of https://github.uci.edu/rtrimana/smart_home_traffic

4 years agoAdding the seconds in which there is 0 packet so that we will see the graph being...
rtrimana [Fri, 1 Dec 2017 23:36:02 +0000 (15:36 -0800)]
Adding the seconds in which there is 0 packet so that we will see the graph being plotted from 0

4 years agoupdate base_gexf_generator with islocal node attribute
Janus Varmarken [Fri, 1 Dec 2017 23:07:24 +0000 (15:07 -0800)]
update base_gexf_generator with islocal node attribute
changes to networkx neighbors call - needs more updates
add flag for producing local-only graphs

4 years agoBipartite generator:
Janus Varmarken [Wed, 29 Nov 2017 06:38:44 +0000 (22:38 -0800)]
Bipartite generator:
- Skip non IP traffic.
- Guard against cases where the device does not perform DNS lookups (or DNS lookups that occurred before data collection started).

4 years agoAdd bipartite graph generator. Needs to be merged with Rahmadi's changes to the base...
Janus Varmarken [Mon, 27 Nov 2017 02:15:54 +0000 (18:15 -0800)]
Add bipartite graph generator. Needs to be merged with Rahmadi's changes to the base generator (I was unable to pull when implementing this functionaliy).

4 years agoAdding binning capabilities to parse_packet_frequency.py script to smoothen and empha...
rtrimana [Fri, 24 Nov 2017 19:43:11 +0000 (11:43 -0800)]
Adding binning capabilities to parse_packet_frequency.py script to smoothen and emphasize certain packet frequencies in the graph

4 years agoAdding packet sizes and send/received bytes plots/analyses
rtrimana [Wed, 15 Nov 2017 18:28:10 +0000 (10:28 -0800)]
Adding packet sizes and send/received bytes plots/analyses

4 years agoAdding new analysis - incoming (not yet including outgoing) packets inter-arrival...
rtrimana [Mon, 13 Nov 2017 19:20:32 +0000 (11:20 -0800)]
Adding new analysis - incoming (not yet including outgoing) packets inter-arrival time

4 years agoAdding traffic volume information into edges
rtrimana [Mon, 13 Nov 2017 18:01:37 +0000 (10:01 -0800)]
Adding traffic volume information into edges

4 years agoCollapsing leaf nodes if they have the same set of protocols
rtrimana [Sat, 11 Nov 2017 00:16:27 +0000 (16:16 -0800)]
Collapsing leaf nodes if they have the same set of protocols

4 years agoAdding protocols as the property of edges; simplifying protocol tracking through...
rtrimana [Fri, 10 Nov 2017 18:29:36 +0000 (10:29 -0800)]
Adding protocols as the property of edges; simplifying protocol tracking through scanning the frame.protocols field

4 years agoAdding excluded devices list; Adding protocol names; restructuring, cleaning-up etc.
rtrimana [Thu, 9 Nov 2017 19:50:23 +0000 (11:50 -0800)]
Adding excluded devices list; Adding protocol names; restructuring, cleaning-up etc.

4 years agoEstablishing basic flow for the complete graph processing
rtrimana [Thu, 9 Nov 2017 16:51:16 +0000 (08:51 -0800)]
Establishing basic flow for the complete graph processing

4 years agoUpdate base_gefx_generator.py to new pipeline
Janus Varmarken [Wed, 8 Nov 2017 23:04:01 +0000 (15:04 -0800)]
Update base_gefx_generator.py to new pipeline

4 years agoSeparating incoming and outgoing traffic for a more fine-grained analysis
rtrimana [Wed, 8 Nov 2017 18:21:43 +0000 (10:21 -0800)]
Separating incoming and outgoing traffic for a more fine-grained analysis

4 years agoAdding moving window average using numpy library
rtrimana [Wed, 8 Nov 2017 00:45:24 +0000 (16:45 -0800)]
Adding moving window average using numpy library

4 years agoAdding moving window average using numpy library
rtrimana [Wed, 8 Nov 2017 00:45:11 +0000 (16:45 -0800)]
Adding moving window average using numpy library

4 years agoCompleting flow for time series graph generation
rtrimana [Tue, 7 Nov 2017 22:26:17 +0000 (14:26 -0800)]
Completing flow for time series graph generation

4 years agoAdding time series analysis run script
rtrimana [Tue, 7 Nov 2017 19:39:20 +0000 (11:39 -0800)]
Adding time series analysis run script

4 years agoSetting device name as attribute (on Gephi we can choose to display/not display it)
rtrimana [Tue, 7 Nov 2017 16:43:29 +0000 (08:43 -0800)]
Setting device name as attribute (on Gephi we can choose to display/not display it)

4 years agoAdding local device name mapping into graph
rtrimana [Tue, 7 Nov 2017 00:44:11 +0000 (16:44 -0800)]
Adding local device name mapping into graph

4 years agoAdding list of MAC addresses for device translation
rtrimana [Mon, 6 Nov 2017 23:40:16 +0000 (15:40 -0800)]
Adding list of MAC addresses for device translation

4 years agoRestructuring files and folders
rtrimana [Mon, 6 Nov 2017 18:49:18 +0000 (10:49 -0800)]
Restructuring files and folders

4 years agoRestructuring files and folders
rtrimana [Mon, 6 Nov 2017 18:24:59 +0000 (10:24 -0800)]
Restructuring files and folders

4 years agoRestructuring files and folders
rtrimana [Mon, 6 Nov 2017 18:12:10 +0000 (10:12 -0800)]
Restructuring files and folders

4 years agoUpdate pipeline: graph now maps IoT devices' MACs to hostnames or other MACs (if...
Janus Varmarken [Mon, 6 Nov 2017 07:45:31 +0000 (23:45 -0800)]
Update pipeline: graph now maps IoT devices' MACs to hostnames or other MACs (if local communication), and device-specific DNS queries are taken into account.
Changes:
- base_gefx_generator.py updated such that it considers MAC addresses instead of IPs and considers device-specific DNS queries for the IoT devices during graph construction.
- extract_from_tshark.py: updated to also include eth.src and eth.dst.
- parse_dns.py: corrected return value of hostname_for_ip_at_time such that the method simply returns the hostname instead of a tuple of timestamp and hostname.
- updated example json generated by extract_from_tshark.py
- added example gephi file generated by base_gefx_generator.py

4 years agoUpdate parse_dns.py with new datastructure that captures all dns requests per device...
Janus Varmarken [Sat, 4 Nov 2017 02:08:58 +0000 (19:08 -0700)]
Update parse_dns.py with new datastructure that captures all dns requests per device [NOTE: needs testing!]

4 years agoDefine DeviceDNSMap: class the stores a specific device's DNS mappings.
Janus Varmarken [Sat, 4 Nov 2017 00:22:33 +0000 (17:22 -0700)]
Define DeviceDNSMap: class the stores a specific device's DNS mappings.

4 years agoInitial experimentation with simple pipeline setup: Call parse_json_dns from parse_dn...
Janus Varmarken [Fri, 3 Nov 2017 19:07:13 +0000 (12:07 -0700)]
Initial experimentation with simple pipeline setup: Call parse_json_dns from parse_dns.py in base_gefx_generator.py
Add simple gitignore.
Add example wireshark DNS extract for testing.
Add example wireshark HTTP extract for testing.

4 years agoFixing indentation and results display
rtrimana [Fri, 3 Nov 2017 15:44:44 +0000 (08:44 -0700)]
Fixing indentation and results display

4 years agoA simple script to analyze the distribution of packet frequencies against time
rtrimana [Fri, 3 Nov 2017 15:33:55 +0000 (08:33 -0700)]
A simple script to analyze the distribution of packet frequencies against time

4 years agoUpdate DNS parser to also construct IP-> hostname map [NOTE: This is the SIMPLEST...
Janus Varmarken [Wed, 1 Nov 2017 20:46:37 +0000 (13:46 -0700)]
Update DNS parser to also construct IP-> hostname map [NOTE: This is the SIMPLEST SOLUTION and does not consider cases where one IP maps to multiple hostnames nor does it consider timing of the mappings -- serves only as a starting point]

4 years agoAdd parse_dns.py: reads DNS traffic from a file (JSON formatted by tshark) and constr...
Janus Varmarken [Fri, 27 Oct 2017 02:46:16 +0000 (19:46 -0700)]
Add parse_dns.py: reads DNS traffic from a file (JSON formatted by tshark) and constructs a map in which a hostname points to a set of IPs associated with that hostname.

4 years agoextract_from_tshark.py:
Janus Varmarken [Thu, 26 Oct 2017 05:00:04 +0000 (22:00 -0700)]
extract_from_tshark.py:
- apply Shuba's fix: do not skip packets that do not have a comment.
- include src ip and src port in output

4 years agoAdd base_gefx_generator.py: script that constructs a .gefx file from JSON generated...
Janus Varmarken [Thu, 26 Oct 2017 04:52:08 +0000 (21:52 -0700)]
Add base_gefx_generator.py: script that constructs a .gefx file from JSON generated by extract_from_tshark.py. The script generates a graph in which nodes are hosts and edges indicate that there is communication between the hosts. The script label/identify hosts by their IPs. It should serve as a starting point when we want to include more information in the graphs (e.g. host name).

4 years agoFirst version of scripts for traffic analysis
Rahmadi Trimananda [Wed, 25 Oct 2017 16:17:04 +0000 (09:17 -0700)]
First version of scripts for traffic analysis

4 years agoInitial commit
Rahmadi Trimananda [Wed, 25 Oct 2017 16:13:36 +0000 (09:13 -0700)]
Initial commit