Carbon Nanotube Eye: Reconstruction of Inner Hidden Composition & Structure of Inspection Targets -Breakthrough in Inspections by “Nano Science” & “Computer Vision”-

Joint Press Release by Chuo University and National Institute of Informatics.

 

Table of Contents
○ Development of non-destructive imaging with a multi-functional photo-sensor into better understandings of hidden compositional identification and structural reconstruction via computer vision^{term 1} technique.
○ Effective use of computer vision in ultrabroad and multi-wavelength bands. Frontier-“invisible photo irradiation-based analysis” leads to breakthroughs in quality assessment techniques.
○ Carbon nanotube-based soft thin-films, a constituent of the photo-sensing eye, facilitate frontier computer vision monitoring by collectively detecting both visible and invisible irradiation in high sensitivities.
○ The presenting device and system allow identifications and reconstructions of composite complicated multi-layer structures concealed by opaque configurations in a non-destructive and non-contact manner.

 

Abstract

  A research group at Chuo University, Japan, organized by Assistant Professor Kou Li (Faculty of Science and Engineering), developed a novel non-destructive inspection technique by effectively combining their own multi-functional photo monitoring device and system with image data-driven three-dimensional (3D) restoration methods, in collaboration with National Institute of Informatics. The presenting technique precisely evaluates target objects by compositional identifications and structural reconstructions (Fig. 1).
  Since the dawn of the IoT (Internet of Things) society, non-destructive inspection techniques have garnered attention to assure the fundamental safety in cyber and physical interactions by detecting concealed defects of daily and industrial components. In particular, photo-imaging methods play core roles in non-destructive inspections for their non-contact and large-area (i.e., informative) data acquisition. Meanwhile, representative inspecting points include compositional identifications (what materials target objects and their concealed defects comprise) and structural reconstructions (how and which shapes the above materials form). Although efforts, collectively satisfying the compositional identifications and structural reconstructions, potentially facilitate precise and reliable social quality testing, studies demonstrating the above concepts are still insufficient. 　
  To tackle such crucial technical limitations, this work developed an unconventional non-destructive inspection platform that would provide a breakthrough in the study field of safety and quality testing. In this work, Assis. Prof. Li, the leader of the collaborative research group, effectively combined the sensing device and system for compositional identifications uniquely developed in Chuo Univ. by employing cutting-edge nanomaterials: Japan-origin carbon nanotubes (CNT), with 3D structural reconstruction schemes in NII: simply based on superposition of silhouette images from target objects. 
  Their original research paper regarding the above topic is available in the international scientific journal “Advanced Optical Materials”, published on December 25^th, 2023. The details are as follows.

【Researchers】

Assis. Prof. Kou Li	Faculty of Science and Engineering, Chuo University
Prof. Yukio  Kawano	Faculty of Science and Engineering, Chuo University
Prof. Imari Sato	National Institute of Informatics

 

【Paper information】

Journal	Advanced Optical Materials
Title	Simple non-destructive and three-dimensional multi-layer visual hull reconstruction with an ultrabroadband carbon nanotubes photo-imager
Author	Kou Li1,†,*, Yuya Kinoshita1,†, Daiki Shikichi1, Miki Kubota1, Norika Takahashi1, Qi Zhang1, Ryo Koshimizu1, Reiji Tadenuma1, Minami Yamamoto1, Leo Takai1, Zhenyu Zhou2, Imari Sato2, and Yukio Kawano1,2,* †Co-first author *Corresponding author
Affiliation	1Department of Electrical, Electronic, and Communication Engineering, Faculty of Science and Engineering, Chuo University 2National Institute of Informatics
DOI	10.1002/adom.202302847

 

【Research content】
1. Background
  While photo monitoring measurements play leading roles in non-destructive inspections, certain bands garner significant attentions: millimeter-wave (MMW), terahertz-wave (THz), and infrared (IR) between radio frequency wave (RF) and visible light (Vis). These wavelength bands collectively satisfy RF-driven permeability to diverse objects and Vis-based straightness (i.e., handling as lasers), enabling visualization of inner parts of targets which are opaque in human eye-sight. Furthermore, broadband and multi-wavelength MMW–IR monitoring measurements lead to compositional identifications since transparency of respective non-metallic materials are widely variable in these bands. Under these circumstances, various works have been focusing on developing inspection devices and systems in MMW–IR bands worldwide, including Assis. Prof. Li. Representative examples of MMW–IR analysis methods (e.g., THz time-domain spectroscopy: THz-TDS^{term 2} and Fourier transform IR spectroscopy: FTIR^{term 3}) now handle food, daily items, and plastic components as quality testing targets.
  From another viewpoint, the research group led by Assis. Prof. Li has developed the design and fabrication of CNT film-based MMW–IR sensors as their unique study field, demonstrating advantageous multi-functional non-destructive inspections^{ref 1–7}. The CNT film sensors comprise flexible and stretchable soft-sheet structures as thin as food wraps, allowing omni-directional attachments to diverse 3D inspecting targets (e.g., gas-liquid pipes and beverage bottles) in 360^o viewing angles without blindspots, contrary to typical rigid imagers (i.e., solid-state cameras). 　
  However, studies synergizing the MMW–IR monitoring measurements with structural reconstruction techniques are still insufficient including spectroscopies and CNT film sensors. In addition to the fact that compositional identifications are essential in quality testing, non-destructive inspections need to collectively satisfy structural reconstructions to monitor both local and entire concealed changes in defect sizes and target shapes induced by external stimulations (e.g., temperature and humidity) in realtime. The representative structural reconstruction technique includes computer vision (CV) methods, which combine optical information (intensities, time delay, and phase differences in reflection/transmission or scattering) with spatial data such as coordinates and angles. Whereas, conventional CV works have been mainly focusing on the use in Vis bands for outer-shape restorations as artistic expressions^ref8, and applications in MMW–IR regions toward inner understandings of various objects are still insufficient. To tackle such crucial technical limitations in respective compositional identifications and structural reconstructions, the research group demonstrates pioneering MMW–IR CV measurements via broadband and multi-wavelength imaging with the CNT film sensor, which makes the most of the advantages in the above bands, as the novel synergetic combinations (Fig. 2).

2. Achievements
  This work employs the photo-thermoelectric (PTE) effect as the operating mechanism of the CNT film sensor. The photo-detection interface is the pn-junction in the center of the CNT film channel, and the device performs subsequential thermoelectric conversion of irradiated energy with the photo-induced heating into electrical readout response signals. Here, the CNT film channel exhibits efficient ultrabroad photo-absorptance (over 90 %) throughout from MMW to IR and further Vis bands. This feature leads to photo-detection with the device in comparable sensitivities with those of typical solid-state sensors, in addition to other advantages: simple all-solution-processable fabrication, ease of handling as compact thin-film soft-sheet configurations, etc. By incorporating these properties, the CNT film PTE sensor performs broadband and multi-wavelength MMW–IR imaging on its own, which plays a key factor in this work.

  Based on these fundamental properties and performances of the device, Assis. Prof. Li organized experimental setups for CV measurements (Fig. 3). As the CV scheme, the research group employed visual-hull method^{term 4}. This method first acquires two-dimensional (2D) silhouette images from different viewpoints, then hollows out the spatially overlapped aera from the reference voxel into simple structural reconstructions. This work therefore organizes photo sources, inspecting objects, and the CNT film PTE sensor, subsequently performs 2D scanning of targets in a xy-plane to obtain silhouette images, and finally conducts θ-rotation toward different viewpoints.

  Assis. Prof. Li further evaluates a composite multi-layer 3D object in the presenting CV system in this work (Fig. 4) to demonstrate the concept of composition-identifying structural reconstructions in non-destructive inspections. The employed target comprises five different materials (semiconductor, glass, plastic, metal, etc.), and its inner is invisible to human eye-sight (i.e., unsuitable for compositional and structural inspections) owing to an opaque outer coating. While daily and industrial components mostly consist of similar configurations with the above target nowadays, the presenting efforts in this work play a benchmarking role for potential social implementations. Under these circumstances, this work succeeded in structural reconstructions per constituent materials of the target via broadband multi-wavelength MMW–IR imaging by the CNT film PTE sensor in an associated combination with the CV scheme of NII to employ each silhouette view (Fig. 5). In detail, THz and IR measurements and the associated silhouette views reconstruct middle- and inner-layers through the opaque outer wall respectively, and the higher permeable MMW monitoring extracts a metallic bar from the deepest center of the target. Meanwhile, the above imaging evaluations compensate each other, since the MMW view is unsuitable for middle-/inner-layers recognitions with its excessive transparency, and the permeability of THz or IR measurements is insufficient against the center housing of the target.

  For the final step, this work non-destructively reconstructs the entire structure of the inspecting target via superposition of respectively extracted 3D models per constituent materials (Fig. 6). This achievement represents the successful application of the fundamental broadband multi-wavelength MMW–IR imaging with the CNT film PTE sensor, unique to Chuo Univ., into the advantageous CV scheme. Proper selections of irradiating wavelengths in the presenting CNT film PTE sensor-based CV system potentially lead to detailed non-destructive inspections of versatile targets based on widely available optical (MMW–IR) databases of constituent materials. Furthermore, multiple pixels-integration^{term 5} of the CNT film PTE sensor serves as another improvement in the presenting device and system: shortening the necessary measurement time. In particular, this work also satisfies the major technical requirements comparable with the cutting-edge existing inspection methods, such as the measurement time of 150 seconds and reconstruction size accuracy within 2 % error by employing a 40 pixels-integrated CNT film PTE array imager.

3. Outlook
  Based on the presenting work, the research group further aims to explore i): filed test in non-destructive inspections against actual daily and industrial components by expanding viewing angle numbers, ii): 2D pixels-integrated camera design with the CNT film PTE sensor toward realtime systemization, and iii): functionality enrichment via deeper understandings of imaging information. While this work currently handles four-cornered column and cylinder structures as inspecting targets from two viewing angles for the proof of concept, demonstrating the first topic (i) would cover diverse objects in uneven surfaces or winding shapes more similar to actual social components in the MMW–IR CV. Moreover, developing integration configurations of the CNT film PTE sensor pixels from the existing one-dimensional arrays into large 2D scale cameras (ii) potentially allows omni-directional device wrapping around targets and spatial scan-less realtime observations. Here, the third topic (iii) is going to enrich CV methods to achieve further informative non-destructive inspections for the CNT film PTE sensor-based MMW–IR imaging such as tomography, photoacoustic, and LiDAR^{term 6}, in addition to the employed visual-hull of this work.

【Reference】
1. K. Li, Y. Matsuzaki, S. Takahara, D. Sakai, Y. Aoshima, N. Takahashi, M. Yamamoto, Y. Kawano, All-Screen-Coatable Photo-Thermoelectric Imagers for Physical and Thermal Durability Enhancement, Advanced Materials Interfaces 10, 35, 2300528, 2023.　DOI: 10.1002/admi.202300528
2. T. Araki, K. Li, D. Suzuki, T. Abe, R. Kawabata, T. Uemura, S. Izumi, S. Tsuruta, N. Terasaki, Y. Kawano, T. Sekitani, Broadband Photodetectors and Imagers in Stretchable Electronics Packaging, Advanced Materials early view, 2304048, 2023.　DOI: 10.1002/adma.202304048
3. K. Li, Y. Kinoshita, D. Sakai, Y. Kawano, Recent Progress in Development of Carbon-Nanotube-Based Photo-Thermoelectric Sensors and Their Applications in Ubiquitous Non-Destructive Inspections, Micromachines 14, 1, 61, 2023.　DOI: 10.3390/mi14010061
4. K. Li, T. Araki, R. Utaki, Y. Tokumoto, M. Sun, S. Yasui, N. Kurihira, Y. Kasai, D. Suzuki, R. Marteijn, J.M.J. Toonder, T. Sekitani, Y. Kawano, Stretchable broadband photo-sensor sheets for nonsampling, source-free, and label-free chemical monitoring by simple deformable wrapping, Science Advances 8, 19, eabm4349, 2022.　DOI: 10.1126/sciadv.abm4349
5. K. Li, R. Yuasa, R. Utaki, M. Sun, Y. Tokumoto, D. Suzuki, Y. Kawano, Robot-assisted, source-camera-coupled multi-view broadband imagers for ubiquitous sensing platform, Nature Communications 12, 3009, 2021.　DOI: 10.1038/s41467-021-23089-w
6. K. Li, D. Suzuki, Y. Kawano, Series Photothermoelectric Coupling Between Two Composite Materials for a Freely Attachable Broadband Imaging Sheet, Advanced Photonics Research 2, 3, 2000095, 2021.　DOI: 10.1002/adpr.20200009
7. D. Suzuki, K. Li, K. Ishibashi, Y. Kawano, A Terahertz Video Camera Patch Sheet with an Adjustable Design based on Self-Aligned, 2D, Suspended Sensor Array Patterning, Advanced Functional Materials 31, 14, 2008931, 2021.　DOI: 10.1002/adfm.2020089318. Shijie Nie, Lin Gu, Yinqiang Zheng, Antony Lam, Nobutaka Ono, Imari Sato, Deeply Learned Filter Response Functions for Hyperspectral Reconstruction, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4767–4776, 2018.
 

【Acknowledgement】 　
  The authors thank Zeon Co. for providing the CNT solutions. The authors also acknowledge Prof. Mariko Matsunaga and Mr. Yuehai Yu in Chuo University for their assistance in performing scanning electron microscopy. This work was financially supported by the ACT-X program (JPMJAX23KL) and the Mirai Program (JPMJMI23G1) of the Japan Science and Technology Agency, JSPS KAKENHI of the Japan Society for the Promotion of Science: JP21H01746, JP21H05809, JP22H01553, JP22H01555, JP22H05470, JP23H00169, JP23K19125, research grant from the Murata Science Foundation (M23 194), research grant from the Matsuo Foundation, research grant from the Sumitomo Electric Groups CSR Foundation, and Strategic Research Development Program of the Kanagawa Institute of Industrial Science and Technology.

【Contact】
About the research:
Kou Li
Assistant Professor, Chuo University, Faculty of Science and Engineering,
(Department of Electrical, Electronic, and Communication Engineering)
TEL: 81-3-3817-1860
E-mail: li★elect.chuo-u.ac.jp
Please replace ★ into @ when sending email.

General Inquiries: 
Chuo University, Research Support Office　
TEL: 81-3-3817-7423 or 1675　FAX: 81-3-3817-1677
E-mail: kkouhou-grp★g.chuo-u.ac.jp
Please replace ★ into @ when sending email.

【Term】
¹ CV: Computer vision  
  CV is a study filed in which information engineering techniques and numerical analyses handle image data for synergetic functionalities.
² THz-TDS: Terahertz Time-Domain Spectroscopy
THz-TDS is one of the representative analysis methods in THz, sub-THz, and MMW bands. THz-TDS is now widely available in the fundamental science study fields.
³ FTIR: Fourier Transform Infrared Spectroscopy
FTIR is one of the representative analysis methods in MIR and FIR bands. FTIR is now widely available in both the fundamental science study and industrial assessments fields.
⁴ Visual-hull
Visual-full is one of the representative CV methods enabling 3D structural reconstructions. The visual-full first acquires 2D silhouette images from different viewpoints and then extracts the overlapped area from the reference voxel as reconstructed 3D models during back-projection.
⁵ Multiple pixels-integration
This technique increases the number of constituent respective sensor pixels in imager devices. In xy-plane silhouette imaging, single-pixel devices require 2D spatial scanning. For employing one-dimensional array imagers with pixels aligning on one axis, scanning inspection targets perpendicular to sensor directions results in obtaining 2D images.
⁶ Tomography, Photoacoustic, LiDAR
These three are some of the representative CV methods together with the aforementioned visual-full. LiDAR stands for “Light Detection and Ranging”. The first one acquires tomographic images, and the photoacoustic measurements allow restorations of vessel structures without external contracting agents. The LiDAR typically serves as vehicle distance sensing and so on, providing structural reconstructions based on such geographic data from the targets.