Net-X-Copy is part of the Net-X-Code Server software. It is designed to convert audio/video files from one type to another and to perform partial file restores from indexed and non indexed files on disk, tape or cloud. This article contains the Linux/macOS/Windows command line API for this tool. A RESTful API is also available.
Net-X-Copy - Partial File Restore
Net-X-Copy supports partial file restore for all the standard broadcast and post production formats we support without the need for a mezzanine or intermediate file format. The original files are indexed, a time code accurate proxy is created, and then any section of the original file can be recalled via time code or absolute offsets based on HTML5 playback of the proxy. The high resolution asset can be stored on near line storage, cloud storage (HTTPS/FTP/Amazon/Google/SMB), tape, or any other kind of block, object, or file storage your server can access.
There are four modes the partial file restore can operate in:
- Full online: This is the simplest mode where access to the source file is fast. Here a call can be made with absolute offset values or time code values to determine the area to restore, and the header/metadata/index in the main file will be used to find the audio/video data and produce the newly wrapped/restored file in the same format.
- Live online: For live files like OpenMXF, transport streams, or other growing files, an area can be clipped or restored while the file is still writing. This includes local, network, and cloud sources. Here the header/metadata is pulled from the file and the index is calculated based on absolute or time code values to find the media for the new file.
- Slow online: For files that are stored on very slow storage, like Amazon's Glacier, or other cold storage systems, an index for the file should first be generated. The file is pushed to long term storage, and the small index file is kept locally available. To restore a clip, the index file is used to calculate the minimum amount of bytes required to create the new file. This byte range can then be retrieved from the file during the restore by Net-X-Copy.
- Full offline. When the file is completely offline, using tapes, stored hard drives, or other methods, an index must be generated before the file is stored. This index can then be used to calculate the minimum amount of the file (as a byte range) that needs to be retrieved to create the restore/wrap file. Once this byte range is available on temporary local storage, the actual restore is called to make the new file.
The index/proxy/restore can be controlled via our RESTful API in www.net-x-code.com or via the command line parameters described below.
Typical Scenarios
This section has some typical proxy, convert and partial file restore workflows. The first scenario is a complete list. The subsequent scenarios assume the first few steps in the first scenario have already been accomplished
Scenario 1 - full access/online source
- The file (MXF, MOV, AVI, CINE, etc) arrives at ingest
- A “command=copy&profile=index” command is sent to index the original filename
- A “command=copy&profile=mp4-h264” command is sent to make a proxy file with time code, multi-track audio, closed captions, metadata and proxy index filename
- Two “command=copy&pisrc&pidst” commands are sent to create JPEG images for the source and proxy files
- At this point, the proxy and main index can be stored in a real or near real time storage, and the main file may be moved to long term storage, tape, cloud (Google/S3) or other offline storage
- The user uses the HTML5 player’s time code (or other time code source) to set one or more In and Out points on the file that needs to be restored
- A “command=copy&profile=wrap” is sent to access the bytes of the original file and create a new file of the same type, without any recompression of audio or video, at the target location
Scenario 2 - tape restore
- This assumes the basic processing in Scenario 1 has been done
- A “command=getcopyinout” is sent with the absolute or time code based in and out points, and the index of source file it will come from
- This returns a series of one or more file names with start and end byte locations
- At this point, the controller restores those byte areas of the files to the name specified by the return
- Once the areas are restored, a “command=copy&profile=wrap” is sent along with the temp folder to create the new output file name
Scenario 3 - cloud restore
- This assumes the basic processing in Scenario 1 has been done
- If the index file is stored on cloud, it can be restored locally first, or read directly from the cloud (HTTPS, FTPS, AWS)
- If the main file is in Glacier, then a command will be sent to restore the section needed to S3 before the restore is done
- Once there is access to the file, or file part, the “command=copy&profile=wrap” can be called normally
- If the resource is in available cloud storage (e.g. not Glacier), then partial file restores may be done from the original file without indexing it first. This will cause more data to be read, but only the headers and tables necessary to find the audio/video/data the restoration needs
Scenario 4 - inline conversion
- This assumes the basic processing in Scenario 1 has been done
- For any restore scenario, the file restored can be a byte accurate re-wrap of the original into a new container, or it can be translated in process (on the fly) to any supported standard format. These formats include MXF Op1a, Op-Atom, P2, IMX, D11, IMF, MOV, Uncompressed and many other containers, with codecs including JPEG-2000, XDCam, MPEG-2, h.264, HEVC, AVCi 100/200, XAVC-S, XAVC, Long-G, TR-01, DV and many others
- The commands can also be used with or without index files to convert all or part(s) of local clips to any of these formats
PFR File Best Practices
Different workflows require different ways of saving and restoring files, but there are some general rules that can make it easier, especially when working with tape or other non sequential storage systems. Media files can be roughly broken down into a few categories:
- Self contained, single files (MXF OP1a, MOV, AVI)
- Multiple stream files (AVI+WAV, Avid OPAtom, MOV QT Reference)
- Multiple files per stream (P2 MXF, XDCam MP4, Canon C300/700 MXF)
- Sequences (DPX, TGA, TIFF)
If you are restoring from reasonable speed, random access devices, then all these types can be simply stored and retrieved as is. If there are sequential access, speed or cost issues, then it makes more sense to make each of these as easily accessible as possible before storing them.
Self Contained
These can be indexed and stored directly.
Multiple Stream Files
These can also be indexed and stored directly, as the RTIN can point at one file per stream. Net-X-Code automatically finds and joins the parts of the streams if it is a supported file type like Avid OPAtom, MOV reference, or video files with rationally named audio. If they are stored this way, there will be one chunk from each file that needs to be restored to make the output file, as the media is in separate files. This is handled in the getinoutbytes return as a series of temp names and start/end byte ranges. If you prefer a single chunk, please follow the guidelines in the Multiple Files Per Stream section.
Multiple Files Per Stream
The RTIN cannot describe streams that have multiple parts per stream. These are normally broken up to get around older disk format restrictions, often at 2 or 4 gigabytes. To deal with these, they should be pre-processed (wrapped) to a self contained file like MXF OP1a before they are stored to tape. Using Net-X-Code to wrap them will cause the original audio/video to be copied to the new MXF without recompression and will automatically generate the RTIN as it is creating the new file. A proxy file can also optionally be created while rewrapping the file. This MXF now becomes the file you would restore from, and that should be stored to tape.
Net-X-Code fully supports automatically joining the parts of most broadcast and post production files for playback and wrapping, including:
- Panasonic P2
- IMF
- AS-02
- DCP
- Canon C300/700
- Sony XDCam MXF
- Sony XDCam MP4
- Avid OP-Atom
- Grass Valley K2 Server Format
- Multi card MXF capture
Avid OP-Atom – Special Case
Net-X-Code supports automatically joining Avid OP-Atom files and creating Avid OP-Atom files that can be directly dropped into the Avid bin for pickup on the next database refresh. If you are using this workflow, then the Multiple Stream Files method works best. If you are importing files into Avid then you will not be able to use OP-Atom, as Avid cannot import even its own. It will want to see the same essence, but in an OP1a container. In this case, you should re-wrap the OP-Atom to OP1a before storing them. It is also possible to restore OP-Atom to OP1a using the “type mxf-op1a” when doing the restore, but if they are stored as OP-Atom you will still need to restore a chunk from each stream to PFR them.
Sequences
As sequences can be restored ‘per file’, they do not need to be indexed.
Sample Command Lines
Basic Conversion Re-Wrap
NetXCopy -s <sourcefile.gxf> -t <targetfile.mxf> -p wrap
Create New Sony XDCam 50
NetXCopy -s <sourcefile.mxf> -t <targetfile.mxf> -p mxf-OP1a-MPEG
Make a Basic Proxy with Audio Mixdown
NetXCopy -s <sourcefile.lxf> -t <proxyfile.mp4> -p mp4-h264 -stereo
Make a Specific Proxy
NetXCopy -s <sourcefile.gxf> -t <proxyfile.mp4> -p mp4-h264 -width 360 -height 202 -h26xprofile main -h26xlevel 41 -gopsize 60 -kilobitrate 800
Make an AS-11 DPP
NetXCopy -s <sourcefile.mov> -t <as11dpp.mxf> -p mxf-as-11-hd-dpp
Make a 10 Bit h.264 OP1a MXF
NetXCopy -s <sourcefile.dpx> -t <mxfop1ah264_10.mxf> -p OP1a_HBR_50
Create An Index For Offline/Tape File
NetXCopy -s <sourcefile.mxf> -p index
Partial File Restore From a Tape
Request the start and end to restore from tape for the actual restore
NetXCopy -s <sourceindex.rtin> -p getCopyInOut -in 01:00:10:00 -out 02:04:30:00 -alignment 4096
This will return (tempFile=sourceindex_147894272_298274816.mxf, a start byte position u64in=147894272, and an end byte position u64out=298274816). Copy the byte section specified from the source= to the tempFile= into your temp directory (d:\record\). Then call the actual restore:
NetXCopy-s <sourceindex.rtin> -t <targetfile.mxf> -p getCopyInOut -in 01:00:10:00 -out 02:04:30:00 -alignment 4096 -tempFolder d:\record\
Replace Audio and Captions in an MXF File
NetXCopy -s <sourcefile.mxf> -t <targetfile.mxf> -p mxf-xdcam-1080i -cc <newlanguage.mcc> -afile <newlanguage.wav>
Make A JPG from a Video Frame
NetXCopy -pisrc <sourcefile.avi> -pidst <sourcefile_100.jpg> -piframe 100
Make a Series Of JPG Frames For Each Second
NetXCopy -pisrc <sourcefile.avi> -pidst <sourcefile_100.jpg> -piskip 30
Write an XMP Metadata File To Disk
NetXCopy -s <sourcefile.avi> -m
Get Clip Info From a Camera Card
NetXCopy -s <path-to-card> -p cardinfo -t 2
Command Line Parameters
netxcopy: Wrap, Convert, Extract, Thumbnail and Index Media Files (v7.1.0.491)
netxcopy -s <sourcefile> -t <targetfile> [-a <ackfile>] -p <profile> [-in <00:01:00:00> -out <00:02:00:00> -fg]
-s <sourcefile> - The source file name and path
-t <targetfile> - The target file name and path
-a <ackfile> - The ACK file name and path. This is the XML acknowledgement file made after a copy
-p metadata - Return XMP metadata for a media file
-p cardinfo - Return XML/JSON info on clips on a camera card
-p dir - Return XML/JSON directory listing
-p clipfiles - Return all the files associated with a media clip
-p md5 - Calculate or check the MD5 value source, send compare string as target
-p <profile> - Profile to use. Current profiles include:
> copy - copy the whole file
> wrap - re wrap file or part of a file
> index - create an RTIndex for a file
> caption - #- Extract captions from a file (see types below)
> cardinfo - cardinfo
> dir - dir
> clipfiles - clipfiles
> md5 - create md5
> xmp - create xmp
> picon - create picon
> metadata - show metadata
> getCopyInOut - get the extents required for a pfr, or use them with a temp file
> getFirstAndLastTimecode - get the first and last time code of a file
> examine - Show file info
> mov-YCbCr8Bit - QuickTime MOV YCbCr 8 Bit
> mov-dvcprohd - QuickTime MOV DVCProHD
> mp4-h264 - QuickTime MOV h264
> mxf-xdcam-720p - MXF OP1a XDCam 720p 50 4:2:2
> mxf-dvcprohd-720p - MXF OP1a DVCPro HD 720p
> mxf-xdcam-1080i - MXF OP1a XDCam 1080i 50 4:2:2
> mxf-dvcprohd-1080i - MXF OP1a DVCPro HD 1080i
> mov-prores422 - QuickTime MOV ProRes 4:2:2
> mov-proreshq - QuickTime MOV ProRes HQ
> mxf-OP1a-MPEG - MXF OP1a MPEG 50 4:2:2
> mxf-OP1a-h264 - MXF OP1a h.264
> mxf-OP1a-HDF - MXF OP1a HDF 50 4:2:2
> mxf-as-11-hd-dpp - MXF AS-11 HD DPP
> scaledown2000k - MP4 - Scale down to 960x540 with a target bitrate of 2mbits
> scaledown500k - MP4 - Scale down to 480x272 with a target bitrate of 500kbits
> mxf-as-11-sd-pal-dpp - MXF AS-11 SD PAL DPP
> mxf-as-11-sd-ntsc-dpp - MXF AS-11 SD NTSC DPP
> hd1080-5000kbs - MP4 - HD 1080 with a target bitrate of 5 mbs
> hd720-2500kbs - MP4 - HD 720p with a target bitrate of 2.5 mbs
> hd360-1250kbs - MP4 - HD 360p with a target bitrate of 1.25 mbs
> h264-7500kbs - MP4 - Any resolution with a target bitrate of 7.5 mbs
> Proxy-h264-5000kbs - MP4 - Any resolution with a target bitrate of 5 mbs
> LBR-h264-10000kbs - MP4 - Any resolution with a target bitrate of 10 mbs
> mxf-OP1a-JPEG2K - MXF OP1a JPEG-2000 4:2:2
> mxf-AS-02-h264-10 - MXF AS-02 h.264 50 mbs
> DASH-MP4-Multibitrate - MP4 - DASH MP4 Multibitrate
> HLS-TS-Multibitrate - TS - HLS Multibitrate
> TS-TR-01-JPEG-2000 - TS TR-01 JPEG-2000
> mxf-OP1a_HBR_50 - MXF OP1a HBR 264 10 bit 50mbs
> mp3-128kbps - MP3 bit rate 128k
> mp4-XAVC-S_4_2_0 - MP4 Sony XAVC-S 4:2:0
> mp4-XAVC-S_4_2_2 - MP4 Sony XAVC-S 4:2:2
> mov-prores444 - QuickTime MOV ProRes 4:4:4
> mov-proresxq - QuickTime MOV ProRes XQ
> aces - ACES Image Files
> dnxhd-mxf-720p - DNxHD 720p 50, 59, 60
> dnxhd-mxf-1080p - DNxHD 1080p 25, 29, 30
> dnxhd-mxf-1080i - DNxHD 1080i 25, 29, 30
> dnxhr-mxf-10-hq - DNxHR 10 Bit
> dnxhr-mxf-sq - DNxHR SQ
> dnxhr-mxf-lq - DNxHR LQ
> TS-MPEG2 - TS-MPEG2 - MPEG-2 4:2:0/passthrough transport stream
> TS-h264 - TS-h264 - h.264 4:2:0/passthrough transport stream
> wave - WAVE audio only
> MXF-RDD-25 - MXF RDD-25 Proxy Writer
> amt3-LowQuality - MXF Avid Media Low Quality Writer
> amt3-StandardQuality - MXF Avid Media Standard Quality Writer
> amt3-HiQuality - MXF Avid Media High Quality Writer
> amt3-HQX_10 - MXF Avid Media HQX 10 Writer
> amt3-DNxHD36 - MXF Avid Media DNxHD 1080 Writer
> MP4-MultiOutput - MP4 - Multiple MP4 Output
> HEVC-h265-10000kbs - MP4 - HEVC Any resolution with a target bitrate of 10 mbs
> mov-xdcam - MOV - MPEG-2 50 4:2:2
> mkv-ffv1 - MKV - FFV1 Lossless
> dnxhr-mxf-hq - DNxHR HQ
> mov-proreslt - QuickTime MOV ProRes LT
> mov-proresproxy - QuickTime MOV ProRes Proxy
> mp4-vtt-h264 - QuickTime MOV h264
> kipro-prores-lt - QuickTime MOV ProRes LT
> amt3-XDCam - MXF Avid Media XDCam Writer
> amt3-Proxy - MXF Avid Media Proxy Writer
> amt3-Uncompressed - MXF Avid Media Uncompressed Writer
> transport_stream - h264 transport stream
> mxf-xdcam35-1080i - MXF OP1a XDCam 1080i 35 4:2:0
> mxf-xdcam35-1080p - MXF OP1a XDCam 1080p 35 4:2:0
> mxf-xdcam35-720p - MXF OP1a XDCam 720p 35 4:2:0
-in <00:01:00:00> - the starting point for the output file in time code or absolute position
-out <00:02:00:00> - the ending point for the output file in time code or absolute position
-absin <200> - the absolute (zero based) start time for the output file (overrides -in)
-tcoffset <00:01:00:00> - Offset the time code by this amount
-tc <01:00:00:00> - Replace the output time code starting with this time code
-ub <ABCD0123> - Replace the output userbits with these userbits
-absout <400> - the absolute (zero based) end time, exclusive, for the output file (overrides -out)
-width <width> - output width of the video (only for arbitrary codecs like h264, hevc and prores)
-height <height> - output height of the video (if not set, the input size or codec size will be used)
-copy - make a copy of the file section we need, instead of reading directly
-dest - folder or folder and file name for the temp file when using copy
-dest <folder> - folder or folder and file name for the temp file when using copy
-cc <mcc/scc file> - replacement closed caption file>
-afile <path-audio-file> - replacement source audio track<s>
-v <path-video-file> - replacement source video track
-stereo - force a stereo pair (mix down) output
-aroute <12345678/auto> - route channels to specific outputs
-uuid <uuid string> - override the UUID of the file with this one
-kilobitrate <kbit rate> - override the kilo bit rate
-h26xprofile <baseline / main / high / high10 / high422 / high444> - override the profile type
-h26xlevel <51> - override the level
-encodemode <0 / 1> - 0 normal, 1 fastest
-gopsize <15> - size of encoded gop
-scenedetect <0> - aggressiveness of scene detection (0-100)
-tempfolder - Temporary folder to store partial file
-alignment - Alignment value for any temporary partial files, for GetCopyInOut profile
-flags flag - Extra flags for special operations
allSubAudioTracks, aiProfileID-##, audiofirstonly, audioforce8channels, ccExtractChannel-##,
ccOutputDirectory-**, convert708to608, use708CompatibiltyBytes, deinterlace-all, deinterlace-auto,
deinterlace-sd, deinterlace-hd1080, forceinterlaced, forceprogressive, deinterlace-motionadaptive,
deinterlace-yadif, deinterlace-edge, deinterlace-triangle, deinterlace-fieldduplication,
eMamAssetID-##, eMamCategory-**, eMamProxyID-##, extractCCLine21, fixOut, forceCCVidStream, forceCaptions,
forceAudioMono, forceAudioStereo, forceAudioBlock, forceOutputFramerate-##, fpsCompensation-##,
frameAccurate, interplay, interpretFileEncoding-##, markFirstTime, multi, nocache, notifyeMam,
offsetTC-##, overrideTC-##, outIsDuration, proxyAfterRecord, overrideextension, proxyDisable, proxyEnable,
proxyRecord, rtinReplaceSource, sampleaccurate, shuffleAudio, testMode, processdiscontinuities,
tctype24/25/df/ndf/50/59/60, timeConversion, timeIsMs, use2channelsonly, zeroBasedOutput
-m - Save the metadata in an XMP file
-fg - force the GUI on
-fc - force command line
NOTE: the parameters in [square brackets] are optional.
For JPEG picons/thumbnails
-pisrc <file-to-picon> - source for the image
-pidst <where-to-make-picon> - target folder and name
-pisize <size> - size of picon, 100%
-piframe <frame-offset-into-file> - frame to use to make the picon
-piskip <number-of-frames-to-skip> - if set, make a picon of each frame at this distance for the whole file
-width <width> - output width of the picon image
For Caption Extraction, these are the numbers to use with -p caption-#
> caption-# - eg caption-0=mcc, caption-6=SCCAUTO, etc
0=MCC, 1=AAF, 2=TEXT, 3=SCC, 4=SCCDF, 5=N0, 6=SCCAUTO,
100=Avid Caption, 101=Belle Nuit Subtitler,
102=CapMaker Plus, 103=Cheetah Caption, 104=Csv, 105=Csv2, 106=Csv3,
107=D-Cine SMPTE 2007, 108=D-Cine SMPTE 2010,
109=EBU STL, 110=Final Cut Pro Xml,
111=Final Cut Xml Gap, 112=Final Cut Pro X Xml,
113=Final Cut Pro XCM, 114=Flash Xml,
115=iTunes Timed Text, 116=JSON, 117=JSON Type 2,
118=JSON Type 3, 119=JSON Type 4, 120=QuickTime text,
121=SAMI, 122=SAMI modern, 123=Scenarist, 124=DVD_SCC,
125=DVD_SCCDF, 126=SoftNi sub, 127=Sony DVDArchitect,
128=Sony DVD Exp dur, 129=Sony DVD line/dur,
130=Sony DVD Tabs, 131=SubRip, 132=Sub Station Alpha,
133=Swift V2, 134=Timed Text 1.0, 135=TT 2006-04 .dfxp,
136=TT 2006-04 .ttml, 137=TT 2006-04 .xml, 138=WebVTT,
139=YT Annotations, 140=YouTube sbv,
141=YouTube Transcript, 142=D-Cinema interop,
143=WebVTT-webvtt, 144=SMPTE-TT 2052,
145=Netflix Timed Text,
-q Quiet mode.
-l Loud mode.
-r 01:00:00:00 Override time code start
-R -00:30:00:00 Offset time codes (e.g. subtract half an hour)
-0 Zero based time code output
-1 Mark this first time code in file
-h # eMAM source clip ID
-s Force CC in video stream
-X Extract CC from line 21 (SD only)
-M # CC channel to extract (def: 1)
-C Use 608 compatibility bytes (not 708)
-c # Convert closed captioning to
-f # Force output frame rate (23, 24, 25, 29, 30)
-g # Compensate conversion (2324, 2423, 2930, 3029)
-o <out-dir> CC output directory
-E # Interpret input file using encoding
0=Auto, 1=UTF-8, 2=UTF-16, 3=IBM-850, 4=IBM-860, 5=Windows-1252
Trademarks, Registered Trademarks, and CopyrightsTrademarks, Registered Trademarks, and Copyrights
Apple Inc. - Apple, the Apple logo, Final Cut, Final Cut Pro, Apple TV, iOS, iPad, iPhone, iPod touch, iTunes, Mac, Mac OS X, macOS, Shake, Final Cut Pro, ProRes, High Sierra, Mojave, M1, M2, and QuickTime are trademarks of Apple Inc., registered in the U.S. and other countries.
Avid Technology, Inc. - Avid Media Composer®, Avid MediaCentral®, Avid Interplay®, and Avid NewsCutter® are either trademarks or registered trademarks of Avid Technology, Inc. or its subsidiaries in the United States and/or other countries.
Belle Nuit Montage - Matthias Bürcher August 2000-2016. All rights reserved. Written in Switzerland. Starting 2016 Belle Nuit Subtitler is released under the GNU Lesser General Public License
CANON KABUSHIKI KAISHA - CANON is a trademark of CANON KABUSHIKI KAISHA
Drastic Technologies, Ltd. – trademarks specified here.
Google LLC – YouTube, Google, Google Cloud, Google.meet.com, and Android are registered trademarks of Google LLC
Grass Valley USA, LLC - Grass Valley®, GV®, the Grass Valley logo, and EDIUS® are trademarks or registered trademarks of Grass Valley USA, LLC, or its affiliated companies in the United States and other jurisdictions.
Linus Torvalds - Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries.
Microsoft Corporation – Microsoft: Windows®, Video For Windows (VFW), DirectShow, Microsoft, Skype, Microsoft Azure, Microsoft Teams, Wave Mapper, Microsoft, Windows NT|2000|XP|XP Professional|Server 2003|Server 2008 |Server 2012, Windows 7, Windows 8, Media Player, Media Encoder, .Net, Internet Explorer, SQL Server 2005|2008|2012|2014, Windows Media Technologies and Internet Explorer are trademarks of Microsoft Corporation.
MPEG LA - MPEG LA licenses patent pools covering essential patents required for use of the MPEG-2, MPEG-4, IEEE 1394, VC-1, ATSC, MVC, MPEG-2 Systems, AVC/H.264 and HEVC standards.
Netflix, Inc. - Netflix is a registered trademark of Netflix, Inc.
Panasonic Holdings Co., Ltd - Panasonic, and Varicam are trademarks of Panasonic Holdings Co., Ltd
Sony Corporation – Sony, Sony DVD Architect, DVD, Catalyst, and Vegas are trademarks of Sony Corporation and/or its affiliates.
All other trademarks are the property of their respective owners.
