ma
/
thesis
1
1
Fork 0
Code Issues 1 Pull Requests Releases 8 Wiki Activity

wörk wörk

master
clemens 2018-05-31 16:29:12 +02:00
parent d3adafda41
commit 885fd770f7
3 changed files with 52 additions and 27 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

56
ThesTeX/content/3-solution.tex
View File

@ -15,16 +15,19 @@ Wait, what did I want to do again?
\end{itemize} \end{itemize}
\section{Evaluating Kibana} \section{Evaluating log processing solutions}
%TODO This chapter looks into the possibilities of existing log processing solutions.
\subsection{Evaluating Kibana} By example, Kibana with an Elastic Search backend and Grafana with an InfluxDB will be evaluated.
To evaluate whether Kibana is a viable approach for the given requirements, I have created a test environment.
This setup is documented in \autoref{app:kibana}.
Two sample datasets were loaded into the Elasticsearch container through HTTP POST requests: \texttt{curl -H 'Content-Type: application/x-ndjson' -XPOST 'elastic:9200/\_bulk?pretty' --data-binary @gamelog.json}.
Once Kibana was told which fields hold the spatial information, it is possible to have a first visualization.
However, this view is optimized for the context of web log processing, so it has a rather low spatial resolution as shown in \autoref{img:kibana} and \autoref{img:kibana2}.
Additionally, the query language restricts the possible research questions the solution can resolve. \subsection{Evaluating Kibana}
To evaluate whether Kibana is a viable approach for the given requirements, a test environmentwas built.
This setup with Docker, definied with Docker-compose, is documented in \autoref{app:kibana}.
Two sample datasets were loaded into the Elasticsearch container through HTTP POST requests. %: \texttt{curl -H 'Content-Type: application/x-ndjson' -XPOST 'elastic:9200/\_bulk?pretty' --data-binary @gamelog.json}.
Once Kibana was told which fields hold the spatial information, it was possible to have a first visualization on the workbench.
However, this view is optimized for the context of web log processing, so it has a rather low spatial resolution as shown in \autoref{img:kibana} and \autoref{img:kibana2}.
Dealing mostly with unprecise locations from GeoIP lookups and in respect of the web users` privacy this choice avoids false conclusions\footnote{GeoIP database providers can not always return qualified resolutions, instead rely on default locations, leadling to bizare events like \url{https://splinternews.com/how-an-internet-mapping-glitch-turned-a-random-kansas-f-1793856052}} and enforces privacy-by-default.
As an additional restraint to application in the geogame context, the query language restricts the possible research questions the solution can resolve.
This means only the questions expressable in the query language can be answered. This means only the questions expressable in the query language can be answered.
Additionally, this requires the users to master the query language before any resonable conclusions can be extracted. Additionally, this requires the users to master the query language before any resonable conclusions can be extracted.
@ -38,13 +41,16 @@ Grafana is a solution to analyze, explore, and visualize various source of time
There exist plugins for nearly any storage and collection backend for metrics\furl{https://grafana.com/plugins?type=datasource}. There exist plugins for nearly any storage and collection backend for metrics\furl{https://grafana.com/plugins?type=datasource}.
The different backends are available through a unified user interface shown in \autoref{img:grafana}. The different backends are available through a unified user interface shown in \autoref{img:grafana}.
Spatial resolution suffers under the same conditions compared to Kibana, with an even lower resolution. %TODO: bild Spatial resolution suffers under similar conditions compared to Kibana, with even lower spatial resolution as shown in \uatoref{img:grafana-spatial}.
\autoref{img:kibana} shows by example the restrictions by the query language/query editing interfaces in the domain of weather stations.
\image{\textwidth}{grafana-spatial}{Spatial resolution in Grafana}{img:grafana-spatial}
\image{\textwidth}{grafana-metrics}{Configuring a graph in Grafana}{img:grafana} \image{\textwidth}{grafana-metrics}{Configuring a graph in Grafana}{img:grafana}
\subsection{Conclusion} \subsection{Conclusion}
After all, the monitoring solutions are no perfect match for this special use case. This chapter once again instantiates the phrase "spatial is special" \cite{spatialspecial}:
After all, the monitoring solutions are no perfect match for this special - spatial - use case.
The privacy concerns vital in web monitoring prohibit detailed spatial analyzes, the query languages can restrict some questions, and custom extensions require constant integration effort. The privacy concerns vital in web monitoring prohibit detailed spatial analyzes, the query languages can restrict some questions, and custom extensions require constant integration effort.
Regarding the specified use cases, expecially the non-expert users benefit from a simple to use interface. Regarding the specified use cases, expecially the non-expert users benefit from a simple to use interface.
@ -75,7 +81,7 @@ The following sections describe each of those components.
The analysis framework takes game logs, processes their entries, collects results, and renders them to an output. The analysis framework takes game logs, processes their entries, collects results, and renders them to an output.
With a Map-Reduce pattern as basic structure for the data flow, an ordered collection of analyzing, matching prostprocessing and render operations defines an analysis run. With a Map-Reduce pattern as basic structure for the data flow, an ordered collection of analyzing, matching prostprocessing and render operations defines an analysis run.
\autoref{img:flow} shows the data flows through the framework. \autoref{img:flow} shows the data flows through the framework.
Every processed log file has its own analysis chain. Every processed log file has its own chain of analyzer instances.
The log entries are fed sequentially into the analysis chain. The log entries are fed sequentially into the analysis chain.
\image{\textwidth}{map-reduce.pdf}{Data flows}{img:flow} \image{\textwidth}{map-reduce.pdf}{Data flows}{img:flow}
@ -102,44 +108,50 @@ This can be useful to filter verbose logs before computationally expensive opera
When all entries of a game log have been processed, the results of each analyzer are collected. When all entries of a game log have been processed, the results of each analyzer are collected.
Each result is linked to the analyzer which has produced this artifact to avoid ambiguous data sets. Each result is linked to the analyzer which has produced this artifact to avoid ambiguous data sets.
The results are stored in a ResultStore.
To support arbitrary structures, a category factory can be specified.
In this case, special analyzers can introduce categories as needed before storing their result.
The newly created category will then be used to store consecutive Analyzer results until another category is introduced.
\subsubsection{Postprocessing \& Render} \subsubsection{Postprocessing \& Render}
When all game logs are processed, the whole result set is passed into the Postprocessing step. When all game logs are processed, the whole result store is passed into the postprocessing step.
This is the first step to compare multiple game logs, i.e. the results of the analyzed game logs, directly. This is the first step to compare multiple game logs, i.e. the results of the analyzed game logs, directly.
Postprocessing is a hard requirement for rendering the results, as at least a transformation into the desired format is absolutley necessary. Postprocessing is a hard requirement for rendering the results, as at least a transformation into the desired output format is absolutely necessary.
Rendering is not restricted to visualizations, artifacts of all kind can be produced. Rendering is not restricted to visualizations, artifacts of all kind can be produced.
A whole range from static plots and CSV exports to structured JSON data for interactive map visualizations is possible. A whole range from static plots and CSV exports to structured JSON data for interactive map visualizations or text generation is possible.
\subsubsection{Log parser} \subsubsection{Log parser}
Key to the framework described above is a component to import game log data, parse it, and prepare it to a common format for processing. Key to the framework described above is a component to import game log data, parse it, and prepare it to a common format for processing.
This needs to be adapted for each supported game. This needs to be adapted for each supported game.
It has to know where game logs are stored and how they can be accessed. It has to know where game logs are stored and how they can be accessed.
Configurable items like URLs allow e.g. different game servers. Configurable items like URLs and user credentials allow e.g. for different game servers.
The important step is the parsing of game logs from the formats used by the games (e.g. JSON, XML, plain text, database, …). The important step is the parsing of game logs from the formats used by the games (e.g. JSON, XML, plain text, database, …) to a common format used internally.
\subsection{Web Interface} \subsection{Web Interface}
The web interface is rather straightforward: The web interface is rather straightforward:
Expert users prepare a set of analysis methods and bundle them with suitable rendering targets to an analysis suit. Expert users prepare a set of analysis methods and bundle them with suitable rendering targets to an analysis suite.
Non-expert users select game logs for processing, choose a prepared analysis suit, and receive a rendered result once the analysis process has finished. Non-expert users select game logs for processing, choose a prepared analysis suit, and receive a rendered result once the analysis process has finished.
\subsection{CLI/API Interface} \subsection{CLI/API Interface}
Providing insight access to analysis and render classes, the CLI/API interface offers the most powerful way to explore the log data. Providing direct access to analysis and render classes, the CLI/API interface offers the most powerful way to explore the log data.
By implementing custom algorithms, expert users can cope with difficult input formats and special requirements. By implementing custom algorithms, expert users can cope with difficult input formats and special requirements.
Splitting often used analysis functionality into small, universal Analyzers, compositing Analyzers into a queue may be sufficient to achieve some information desires.
\subsection{Architecture} \subsection{Architecture}
The API is a standalone microservice. The API is designed to be standalone, i.e. it is independent of both game servers and user interfaces.
It is independent of both game servers and user interfaces.
Separation from game servers narrows the scope, and allows the usage with any kind of game. Separation from game servers narrows the scope, and allows the usage with any kind of game.
Games without central server can provide a mocked server to supply logged data, while games with server can e.g. expose API endpoints with authentication and user management. Games without central server can provide a mocked server to supply logged data, while games with server can e.g. expose API endpoints with authentication and user management.
By acting like any normal client, the framework can avoid obstacles like CORS/XSS prevention. By acting like any normal client, the framework can avoid obstacles like CORS/XSS prevention.
The independence to user interfaces, mainly the web interface, allows scalability through load-balancing with mulitple API workers. The independence to user interfaces, mainly the web interface, allows scalability through load-balancing with mulitple API workers.
Expert users with special requirements can embed the framework in projects without pulling in large amounts of dependencies. Expert users with special requirements can embed the framework in projects without pulling in large amounts of dependencies for user interfaces or games/game servers.
\begin{comment} \begin{comment}

18
ThesTeX/content/4-implementation.tex
View File

@ -1,9 +1,15 @@
Based on the findings in \autoref{sec:solution}, an implementation with Python was realized.
\section{klassenstruktur} \section{klassenstruktur}
\section{konfigurierbarkeit (log import)} \subsection{Analysis Framework}
\section{...}
\image{\textwidth}{architecture.pdf}{archoitecure overview}{img:arch} \subsection{Web Interface}
\subsection{Integration \& Service composition}
%TODO: end
\section{1. vortrag} \section{1. vortrag}
\section{Outlook: Implementation} \section{Outlook: Implementation}
@ -37,11 +43,13 @@ Rendering
Configuration \& results Configuration \& results
\twofigures{0.5}{../../PresTeX/images/oeb-kml}{Analyzer configuration}{img:oebkml}{../../PresTeX/images/oeb-ge}{Result visualized}{img:oebge}{Example: Generate KML tracks (BioDiv2Go; Oberelsbach2016)}{fig:oeb2016} \twofigures{0.5}{../../PresTeX/images/oeb-kml}{Analyzer configuration}{img:oebkml}{../../PresTeX/images/oeb-ge}{Result visualized}{img:oebge}{Example: Generate KML tracks (BioDiv2Go; Oberelsbach2016)}{fig:oeb2016}
{ActivityMapper} ActivityMapper
\image{.7\textwidth}{../../PresTeX/images/track-fi}{Combined screen activity and spatial progress}{img:trackfi} \image{.7\textwidth}{../../PresTeX/images/track-fi}{Combined screen activity and spatial progress}{img:trackfi}
{Graphs} Graphs
\image{\textwidth}{../../PresTeX/images/simu-retries}{Experimentational rounds}{img:retries} \image{\textwidth}{../../PresTeX/images/simu-retries}{Experimentational rounds}{img:retries}
\image{\textwidth}{../../PresTeX/images/speed}{Speed distribution}{img:speed} \image{\textwidth}{../../PresTeX/images/speed}{Speed distribution}{img:speed}
\image{.9\textwidth}{../../PresTeX/images/time-rel}{Time distribution}{img:time} \image{.9\textwidth}{../../PresTeX/images/time-rel}{Time distribution}{img:time}
\image{\textwidth}{architecture.pdf}{archoitecure overview}{img:arch}

5
ThesTeX/content/literature.bib
View File

@ -535,3 +535,8 @@ address="Berlin, Heidelberg",
pages="496--510", pages="496--510",
isbn="978-3-642-23199-5" isbn="978-3-642-23199-5"
} }
@article{spatialspecial,
title={What is Special About Spatial Data? Alternative Perspectives on Spatial Data Analysis (89-4)},
author={Anselin, Luc},
year={1989}
}