The engraving is at least 430,000 years old, meaning it was done by the long-extinct Homo erectus, said the study.
The oldest man-made markings previously found were about 130,000 years old.
If confirmed, experts say the findings published in the journal Nature may force a rethink of how human culture developed.
One of the report's authors, Stephen Munro, told the BBC it could "rewrite human history".
"This is the first time we have found evidence for Homo erectus behaving this way," said the researcher, from Australian National University.
'No other explanation'Hundreds of fossilised freshwater mussel shells were excavated and collected in Java by Dutch scientist Eugene Dubois in the 1890s, then stored in boxes for years in the Dutch city of Leiden.
In May 2007, Mr Munro took photos of them as part of his research for his PhD.
The engravings stood out very clearly on the digital photos when they had not been visible to the naked eye.
Scientists found the engravings were as old as the shell, made more than 430,000 year ago
"Immediately when I saw the markings there I thought, those are human engravings, there's no other explanation," he said.
A group of scientists then worked to date the shell and check whether the engravings were as old as the shell itself.
They found the engravings were indeed made before fossilisation, when the mussel was fresh between 430,000 and 540,000 years ago.
Mr Munro said the discovery could confirm theories that Homo erectus had significant manual dexterity and greater cognitive abilities than previously thought.
"We see this type of behaviour, whether it's art or symbolic expressions, we reserve that behaviour for ourselves. As something quite uniquely human," he said.
"With this finding, we might say there are definitely difference between us and Homo erectus. But they might be more like us than we previously thought."
It is not clear whether the pattern was a form of art, or served another purpose.
Other experts expressed scepticism about the research.
John Shea, from Stony Brook University in New York, told NPR there was "nothing like it around for hundreds of thousands of years, and thousands and thousands of miles".
"If this is symbolic behaviour by Homo erectus, then it's basically the only evidence we've got for a species that lived for a million-and-a-half years on three continents," he said.