|
October 6, 2000
JASRAC
Japanese Society for Rights of
Authors, Composers and Publishers
Nomura Research Institute, Ltd.
Announcement of Evaluation Test Results for “STEP 2000”, International
Evaluation Project for Digital Watermark Technology for Music
The Japanese Society for Rights of Authors, Composers and Publishers (Hereunder
referred to as JASRAC. Headquarters: Shibuya-ward, Tokyo. President: Mr. Shigeru
Yoshida.), with Nomura Research Institute, Ltd. (Hereunder referred to as NRI.
Headquarters: Chiyoda-ward, Tokyo. President: Mr. Shozo Hashimoto) commissioned
to provide services, together with international associations of copyright management
societies, CISAC (Headquarters: Paris. Secretary General: Mr. Eric Baptiste) and
BIEM (Headquarters: Paris. Secretary General: Mr. Ronald Mooij) have completed
their joint international evaluation project for audio digital watermarking technology,
the first of its kind initiated by copyright management bodies, “STEP 2000”.
JASRAC, which served as the project leader, has completed its technology evaluation certification based on the final report submitted by the NRI, which served as the project secretariat, and hereby announces its conclusions.
Project Overview
The STEP 2000 project was inaugurated on June 15, 2000. The application deadline
of July 14, the submission of evaluation material of August 14 and the completion
of evaluation testing on September 14 were all carried out on schedule. STEP 2000’
aim “to certify the aptitude of digital watermark technologies, with a view
towards promoting its utilization” met with enthusiastic responses from many
technology enterprises. Participation applications were received from enterprises
in Europe, the US and Asia, and an extensive technology evaluation was conducted.
Technology Evaluation Focus
The evaluation (testing) of submitted digital watermark technologies was conducted
with the following focuses.
1) Robustness: Whether the watermarked data can be extracted after the various
processes of music usage.
2) Audibility: Whether Golden Ears and Silver Ears can detect whether data watermarks
have been embedded in music that is played back in a recording studio environment.
In addition to the above, the evaluation also took into consideration acoustic
analysis results and time elapsed in embedding the watermark and extracting watermarked
information. The watermarked music that was submitted for evaluation included,
as requirements as digital watermark data, 2 bit Copy Control Information (CCI)
within a timeframe of 15 seconds, and 72 bit Copyright Management Information
(CMI) within a timeframe of 30 seconds.
*Notes on the Evaluation
For digital watermarking technologies, robustness and audibility are in a tradeoff
relationship, as are other requirements in attaining robustness. Aside from the
capabilities these technologies have, in their utilization, it is necessary to
tune the technology so that balance is attained in meeting these numerous requirements.
STEP 2000 evaluated technologies that were submitted before the set deadline,
and has not given the participants additional opportunities to tune their technologies
to meet evaluation criteria for the various requirements. Therefore, consideration
was granted in the evaluation to technologies whose performances for certain requirements
did not meet evaluation criteria, but were assumed to be capable of meeting those
criteria upon conducting proper tuning.
Evaluation Results (Certified Companies)
The companies STEP 2000 has certified as possessing technologies that may be recommended
to businesses involved in the digital distribution of music are as follows.
| l |
IBM |
| l |
MarkAny (Korea) *1 |
| l |
Victor Company of Japan *2 |
| l |
Signum (England) *2 |
| l |
BlueSpike (USA) *3 |
| *1) |
The evaluated technology has an
optimum balance between audibility and robustness, however, desires tuning that
caters to the increase of the rate of persistence of the CMI. |
| *2) |
The evaluated technology desires further tuning
balance between audibility and robustness. |
| *3) |
The evaluated technology is weighted heavily on audibility,
and desires tuning that caters to the improvement of overall robustness |
For questions concerning STEP 2000, please contact:
JASRAC Public Relations Department: Mr.Hayashi
Tel.(03)3481-2164,
E-mail:koho@pop02.jasrac.or.jp
URL https://www.jasrac.or.jp
NRI Corporate Communications Department: Kenji Taniguchi, Takeshi Tokushige
Tel.(03)5255-1981, E-mail:kouhou@nri.co.jp
[Annex 1] Overview of the Evaluation Test
The certified enterprises are enterprises that submitted technologies that performed
outstandingly in evaluation tests under the following conditions.
1. Musical Works Used for the Tests
The musical works used as the assignments were recorded on a CD-R as a WAV format
file (44.1kHz/16bit/2ch) and distributed to participating enterprises. The robustness
test was conducted on 2 popular music compositions, 1 classical composition, and
2 compositions with distinctive acoustics. The sound quality test was conducted
on one of the compositions from each of the aforementioned categories, namely
popular music, classical and distinctive acoustic. (The distinctive acoustic compositions
used as assignments were traditional Japanese music and percussion compositions.)
2. Overview of the Robustness Tests
The robustness test evaluated whether the data watermarked by the participating
enterprises could be extracted after the following processes, in the following
environment, under equal conditions for all submitted technologies.
| l |
Processing in a mastering studio |
| l |
Processing in a broadcasting studio (and a prospective broadcasting environment) |
| l |
Processing for distribution through the Internet and other networks |
| l |
Processing using commonly available consumer level equipment |
| The specific robustness testing items are as follows. |
Robustness testing items
| Testing Item |
Overview of Processing Involved |
| D/A, A/D transition |
Digital » Analog » Digital |
| Altered number of channels |
Stereo (2ch) » mono |
| Down sampling |
44.1kHz/16bit/2ch » 16kHz/16bit/2ch |
| Amplitude compression |
44.1kHz/16bit/2ch » 44.1kHz/8bit/2ch |
| Time and pitch compression and decompression |
l Time
compression / decompression: ± 10%
lPitch shift
compression / decompression: ± 10% |
| Linear data compression |
lMPEG
1 Audio Layer 3 (MP3): 128kbps
lMPEG 2 AAC: 128kbps
lATRAC: Version 4.5
lATRAC 3: 105kbps
lRealAudio: ISDN
lWindows
Media Audio: ISDN |
| Non-linear data compression |
lFM
(FM multiple broadcast, terrestrial hertzian TV broadcast)
lAM
(AM broadcast)
lPCM
(Satellite TV broadcast: communications satellite, broadcasting satellite) |
| Characteristic transformation of frequency response |
lFM
(FM multiple broadcast, terrestrial hertzian TV broadcast)
lAM (AM broadcast)
lPCM
(Satellite TV broadcast: communications satellite, broadcasting satellite) |
| Noise |
White noise: S/N: - 40dB |
|
3. Overview of Audibility
| 1) |
ABX test conducted |
| l |
The listener listens to a sound recording
with no watermark (A), a sound recording with watermarks embedded (B), and a sound
recording which is one of the two (X). |
| l |
The listener listens to A and B alternately
twice for 40 seconds each, and then listens to X for 40 seconds. The listener
decides whether X is A or B. |
| 2) |
Elimination of (correct) contingency
responses |
| l |
In order to eliminate (correct) contingency
responses, the above tests were conducted 5 times for each participating enterprise. |
| l |
The listener is defined to have detected
the embedded watermark if the same listener correctly determines whether the watermark
is embedded or not on each of the 5 tests.Ö Under this definition, the significance
of the responses are 95% or greater. |
| 3) |
4 individuals each from the following occupation
categories were selected as Golden Ears and Silver Ears from the recording industry. |
| l |
Recording engineers |
| l |
Mastering engineers |
| l |
Synthesizer manipulators |
| l |
Audio critics |
[Annex 2] Summary of Evaluation Test Results
1. Summary of Robustness Test Results
| l |
Several enterprises scored highly
on robustness tests for both CCI and CMI. This outcome, and the results of the
audibility tests indicate that the technology level of digital watermarks for
music have moved beyond the research and experiment stage and have reached a level
of practical utilization. |
| l |
Many participating enterprises faired
well in “D/A – A/D transition”, which is a security requirement
peculiar to digital watermarks, and one of the most important robustness requirements.
On the other hand, for “down sampling”, “amplitude compression”,
“time and pitch compression / decompression” and “altered number
of channels”, the results were polarized as some enterprises scored highly
while others did not. |
| l |
Many participating enterprises faired well in
tests on overall robustness against compression. However, results for tests on
robustness against high levels of compression (especially for “RealAudio”
which is a streaming transmission) were generally poor. |
| l |
Aside from a few exceptions, good results could
not be obtained on tests for robustness against compression for broadcasting in
a broadcasting environment for AM broadcasting. However, for FM and PCM broadcasting,
several enterprises scored highly. |
2. Summary of Audibility Test Results
| l |
Watermarks from several enterprises
went undetected by many of the Golden Ears and Silver Ears for all of the assignment
compositions. |
| l |
The level of difficulty in embedding
watermarks, when judged from the rate of detection by the listeners, was greatest
for distinctive acoustic compositions, then classical music, and finally popular
music. The level of detection in popular music for enterprises that received certification
was extremely low. |
| l |
Enterprises that scored highly in audibility
tests were also proven capable in acoustic science analysis (frequency analysis,
etc.) as well. |
3. Others (Processing Time)
| l |
Differences in time required to
embed watermarks per one minute of music were insignificant. However, for the
extracting of the watermarks, some enterprises required only several seconds to
process one minute of music, while others required much more time. |
|
