Doesn't this mean we have to load every triple in RAM in order to de-dupe, or is there something clever happening under the hood?

Why do we need to de-dupe? (And yes, @cbeer, putting these triples in an ArrayList or LinkedHashSet will pull them all into memory. Pretty much anything from the basic Java Collection API that impls Collection is an in-memory construct.)

I think we should de-dupe the rdf:type statements -- it looks sloppy to repeat them.

Ideally, we would be able to avoid adding the duplicate types in the first place. But if that doesn't work, then a better pattern would be an Iterator implementation that wraps the triples and suppresses duplicate rdf:type triples. We would only need the rdf:type triples in memory, which would avoid the typical cases where this would be a problem (e.g. large numbers of children, etc.).

It would be much better to do some kind of limited filtering. The extant filter() method could be reused for this. The whole purpose of RdfStream was originally precisely to avoid pulling all the triples into heap, because we saw severe performance degradations when that happened.

@cbeer, @ajs6f, are you ok with duplicate triples? Should we find another technique for maintaining uniqueness or should this ticket be dropped?

I don't think we should merge this as is. It would be good to understand why there are duplicate triples in the first place. It'd surprise me if that was necessary.

I believe the ticket describes why (or rather "when") the duplicate triples are introduced.
https://www.pivotaltracker.com/story/show/71751536

The idea of not introducing duplicates in the first place is appealing, as it is not clear to me how we would filter duplicates without building up the triples in-memory.

1. I'm actually fine with duplicate triples. I understand @escowles's point, but it just doesn't bother me that much. RDF is a machine format.

2. If we can understand how to avoid the duplicates at some closer-to-the-JCR layer, that is without question the right thing to do. It shouldn't be that hard. It currently seems to be a matter of maintaining some in-flight state in the subclasses of NodeRdfContext.

3. NodeRdfContext is a piece of junk. In fact, the whole triple generation subsystem should be reworked to bring far more of the action out into the type system. And yes, I wrote a huge amount of that and that's one reason I'm so sure about it.