Previously, I published in a separate post a minimal example of some Java code that reads WARC data, using the "jwat-warc" library. After that, I wanted to compare the performance of the two major WARC parsing libraries ("jwat-warc" and "jwarc"), in the context of a more complex process.
The difficulty in doing this is that the major libraries have different approaches to how they expect you to consume the data.
e.g.
- "jwat-warc" exposes a stream of the underlying HTML, while "jwarc" exposes a stream containing both the underlying HTTP headers and the HTML.
- "jwat-warc" exposes the HTTP response code, while when using "jwarc" they have to be extracted from the headers.
- etc.
Here is the interface I came up with that represents a fully loaded WARC record, independent of the underlying implementation, alongside two static utility functions for constructing an XWarcRecord instance from the library-specific WarcRecord instance:
public abstract class XWarcRecord {
protected String _uri;
protected String _payload;
abstract public String responseCode();
abstract public String payload();
abstract public String uri();
public static XWarcRecord from(org.jwat.warc.WarcRecord r) {
return new XWarcRecord_JWat(r);
}
public static XWarcRecord from(org.netpreserve.jwarc.WarcRecord r) {
return new XWarcRecord_JWarc(r);
}
}
And the particular implementations, with some optimizations to defer as much processing as possible to the time when a particular data point is needed:
I. "jwat-warc"
For some reason, if you don't read the stream manually, the reader will throw an error when advancing to the next record. So we have to do that in the constructor.
public class XWarcRecord_JWat extends XWarcRecord {
private final WarcRecord r;
XWarcRecord_JWat(WarcRecord r) {
this.r = r;
try {
InputStream contentStream = r.getPayloadContent();
this._payload = new String(contentStream.readAllBytes(), Charsets.UTF_8);
} catch (IOException | NullPointerException e) {
this._payload = null;
}
}
@Override
public String responseCode() {
if (r.getHttpHeader() == null) {
return "0";
}
return r.getHttpHeader().statusCodeStr;
}
@Override
public String payload() {
return _payload;
}
@Override
public String uri() {
if (_uri == null) {
_uri = r.getHeader("WARC-Target-URI").value;
}
return _uri;
}
}
II. "jwarc"
The stream they expose includes both the HTTP headers and the HTML (or other content), so we have to extract them manually.
public class XWarcRecord_JWarc extends XWarcRecord {
protected String _headers = null;
protected String _responseCode = null;
private final WarcRecord r;
XWarcRecord_JWarc(WarcRecord r) {
this.r = r;
}
@Override
public String responseCode() {
if (_responseCode == null) {
_parseContent();
_responseCode = _responseCode(this._headers);
}
return _responseCode;
}
@Override
public String payload() {
if (this._payload == null) {
this._parseContent();
}
return this._payload;
}
@Override
public String uri() {
if (_uri == null) {
_uri = r.headers().first("WARC-Target-URI").orElse(null);
}
return _uri;
}
private void _parseContent() {
List<String> h = new ArrayList<>();
List<String> p = new ArrayList<>();
try (MessageBody body = r.body();
BufferedReader reader = new BufferedReader(new InputStreamReader(body.stream(), StandardCharsets.UTF_8))) {
String line;
while ((line = reader.readLine()) != null) {
if (line.isEmpty()) {
break;
}
h.add(line);
}
while ((line = reader.readLine()) != null) {
p.add(line);
}
this._headers = String.join("\n", h);
this._payload = String.join("\n", p);
} catch (IOException | NullPointerException e) {
throw new RuntimeException(e);
}
}
private static final Pattern P = Pattern.compile("HTTP/\\d\\.\\d\\s+(\\d{3})");
private static String _responseCode(String input) {
Matcher matcher = P.matcher(input);
if (matcher.find()) {
return matcher.group(1);
} else {
return "0";
}
}
}
There was no observed performance difference between the two, but coming up with a solution to abstract away the underlying implementation was an interesting exercise.